greatitself.
Continue
advanced25 min studyLesson 383 in Engineering

Canary analysis and guardrails

Limit exposure with explicit health criteria while accounting for sample size, comparison bias, repeated decisions, and missing telemetry.

Before you begin
On this page

The question for today

What you’ll understand

  • Separate exposure control from evidence of comparable outcomes.
  • Calculate a simple error-rate difference and its uncertainty model.
  • Explain multiple-look and selection-bias failures in automatic promotion rules.

A canary deliberately exposes a candidate to a limited population before wider release. The mechanism reduces the scope of some defects; the analysis decides what the observations support. These are different controls. A tiny healthy-looking sample, an unrepresentative cohort, or missing telemetry should not be silently interpreted as evidence that a release is safe for every user and workload.

Define the population and the stop condition first

Choose which traffic or users can enter the canary, how assignment persists, and which baseline is comparable. Routing weights do not always equal observed request fractions because sessions, retries, geography, and request sizes differ. A replica count is also not necessarily a traffic percentage. Record the actual exposure and the cohort composition. Metrics should cover user-visible correctness and latency as well as resource health; a successful HTTP response that returns a wrong recommendation or loses a payment is not a good outcome simply because its status code is 200.

Predefine immediate guardrails for severe failures and a separate statistical plan for more subtle changes. Set minimum observations, a maximum exposure or duration, treatment of stale or missing metrics, and an inconclusive path. A monitor query that returns no data is not automatically a zero error rate. A canary that shares an overloaded dependency with baseline can affect both groups, and sticky assignment can correlate requests. Compare like populations and reason about interference before interpreting a difference as a causal effect of the candidate.

Δ^=p^c−p^b,p^j=ejnj\widehat\Delta=\widehat p_c-\widehat p_b,\qquad \widehat p_j=\frac{e_j}{n_j}
For positive counted eligible requests n_j and error counts e_j, p̂_j is an observed error fraction and Δ̂ is a difference in fractions. c denotes candidate and b baseline. Causal interpretation requires comparable assignment, consistent outcomes, and appropriate handling of dependence and interference.↔ Scroll wide equations horizontally.

Uncertainty depends on effective observations

Under independent Bernoulli outcomes in two independent groups, an approximate standard error for the difference comes from adding the two sample-proportion variances. Small error counts, rare events, clustered users, repeated sessions, and changing conditions can make that approximation inappropriate. Use an analysis method matched to the design instead of treating a normal interval as automatic truth. Increasing raw request count does not create independent users when one client sends thousands of correlated requests. A candidate with fewer samples also contributes more uncertainty even when its displayed percentage resembles baseline.

Repeatedly checking many metrics and windows creates more chances for an accidental alarming result or a conveniently selected pass. A predeclared finite family can use valid per-comparison controls and a union-bound allocation; sequential methods offer other designs. Such controls do not cure selection bias, invalid individual tests, or metric definitions changed after viewing the outcome. Promotion rules should preserve the analysis version and all decision looks, including inconclusive or unfavorable ones, so a later reviewer can reconstruct why exposure increased.

SE⁡^(Δ^)=p^c(1−p^c)nc+p^b(1−p^b)nb\widehat{\operatorname{SE}}(\widehat\Delta)=\sqrt{\frac{\widehat p_c(1-\widehat p_c)}{n_c}+\frac{\widehat p_b(1-\widehat p_b)}{n_b}}
This approximate standard error is dimensionless and assumes independent Bernoulli observations within and between the two groups, with sufficiently informative counts for the intended normal approximation. It does not account for clustering, adaptive assignment, repeated looks, or interference.↔ Scroll wide equations horizontally.

Automate the specified decision, including uncertainty

Argo Rollouts and Flagger can evaluate configured measurements and drive progress or abort actions. Their presence does not validate the statistical design of every expression supplied to them. Require evidence freshness, correct release labeling, comparable windows, and a deliberate treatment of query errors or empty results. A hard latency or correctness guardrail may stop exposure without waiting for a significance calculation. A failure to reject a difference is not proof of equivalence; if equivalence or noninferiority is the requirement, define its margin and suitable evidence explicitly.

The sampling instrument shows only one independent-proportion model. Reset, set both p=0.1 and n=20, predict sample-mean variance 0.0045, then Play or Step. Separately Reset, set both p=0.1 and n=80, predict variance 0.001125, then Play or Step. Exact binomial bars describe a supplied common probability, and playback highlights outcomes instead of collecting data. The normal comparison can be poor for rare counts. This is not a canary-analysis engine, sequential test, or causal estimator; identify the real cohort and telemetry assumptions it omits.

Work it through

Count errors before interpreting percentages

A fixed comparable-window exercise observes 100 errors in 5,000 candidate requests and 200 errors in 20,000 baseline requests. For a teaching uncertainty calculation, assume independent Bernoulli outcomes and independent groups.

  1. Candidate error fraction is 100/5,000=0.02 and baseline is 200/20,000=0.01. Their difference is 0.01, or one percentage point; the candidate rate is twice baseline, not one percent relatively higher.

  2. The estimated variance is 0.02×0.98/5,000 + 0.01×0.99/20,000 = 0.000004415. Its square root is approximately 0.002101, or 0.2101 percentage points.

  3. At a counterfactual baseline fraction of 0.01 for those same 5,000 candidate requests, expected baseline errors would be fifty, compared with the observed hundred. That fifty-error contrast is descriptive unless the comparable-assignment assumptions support the causal interpretation; repeated-look policy and severe guardrails remain separate.

The result

The data show a concrete observed difference under a declared scope. They do not justify bypassing the preregistered decision design or ignoring cohort imbalance.

The living notebook

Explore the model

Interactive lab

Interactive lab · sampling

Sample size changes one uncertainty term

Reset, set both p=0.1 and n=20, predict sample-mean variance 0.0045, then Play or Step. Separately Reset, set both p=0.1 and n=80, predict 0.001125, then Play or Step. Exact independent Bernoulli probabilities neither collect canary data nor handle sequential looks, assignment bias, or clustered requests.

Sample size changes one uncertainty termExact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Current readings: Mean sample proportion: 0.5; Sample-mean variance: 0.0125; Uniform CDF error bound: ≤ 0.1062. All parameters are adjustable with the labeled controls below.Exact distribution of the mean of 20 independent Bernoulli variables with success probability 0.5; playback highlights outcome 0.00.250.50.75100.0580.1150.1730.23Sample mean · k / nProbability massNormal bin masses are an approximationk = 0 · exact probability = 9.54e-7
Exact binomial massNormal interval massHighlighted outcome / mean
Mean sample proportion
0.5
Sample-mean variance
0.0125
Uniform CDF error bound
≤ 0.1062
t = 0 s

Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forMore observations reduce this modeled variance only under its assumptions; they do not repair a biased comparison or an invalid promotion rule.

Pause & reason

Check your understanding

A configured canary metric query returns no samples. Which response best preserves the meaning of the release evidence?

Your turn

Try it yourself

A predeclared family has twelve metrics examined at four fixed decision looks. Using a simple union-bound allocation and valid individual tests, what per-comparison false-alarm level limits the family false-alarm probability to at most 0.05?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Count forty-eight comparisons and divide the allowed total probability among them.

Compare with the solution

Allocate 0.05/48≈0.00104167 to each comparison. The union bound does not require independence between comparisons, but the individual error guarantees and predeclared family must be valid. This arithmetic does not fix adaptive unrecorded looks, selection bias, correlated-observation misuse, or a mismatched null hypothesis.

Keep the idea

Take it with you

  • Limit exposure and validate the comparison design as separate controls.
  • Count effective observations and preserve the scope of rate differences.
  • Missing data, repeated looks, and inconclusive evidence need explicit decision rules.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.