advanced32 min studyLesson 38 in Engineering

Progressive delivery

Expose a release in measured stages with capacity-aware traffic allocation, comparable observations, and reversible promotion gates.

On this page

What you’ll understand

  • Distinguish a replica fraction from an accurately controlled traffic fraction.
  • Calculate canary load, sample size, and expected error counts.
  • Set promotion and abort criteria that account for delayed and missing evidence.

A deployment can replace every process successfully while making the product worse. Progressive delivery narrows the first exposure and asks whether the new version behaves acceptably before widening it. The key work is designing a meaningful comparison and a trustworthy abort path. A small canary with unrepresentative traffic or insufficient observations is not automatically safer; it can merely delay discovering a problem until promotion exposes everyone.

Control exposure independently of warm capacity

A basic canary can approximate exposure by changing the ratio of old and new replicas. With a small fleet, integer replica counts limit that approximation, and persistent connections can skew actual request shares. Argo Rollouts supports explicit canary steps and traffic-routing integrations for finer control. When routing and replica scaling are decoupled, ensure that the chosen number of canary replicas can handle the assigned traffic. A traffic weight is not a resource reservation.

Keep rollback capacity available when the recovery objective requires an immediate traffic shift. Running both versions may cost more temporarily, but terminating the old fleet too early can turn an abort into a slow scale-up. Determine whether the new version changes shared data, emitted events, or configuration in ways that the old version cannot understand. Traffic can be redirected quickly while an incompatible database mutation remains irreversible.

λc=wλ,mc≥⌈λcμreplica⌉\lambda_c=w\lambda,\qquad m_c\geq\left\lceil\frac{\lambda_c}{\mu_{\mathrm{replica}}}\right\rceil
w is the actual canary traffic fraction, λ is total requests/s, and μ is measured capacity per canary replica at the objective. The lower bound contains no safety headroom and assumes balanced comparable requests.↔ Scroll wide equations horizontally.

Compare cohorts and define evidence before promotion

Choose metrics from user-visible behavior: successful completion, latency, correctness, and resource cost can all matter. Compare compatible traffic cohorts over the same interval. Sending only employees to the canary can be useful for early smoke testing but may miss production payloads, geographic effects, or account-specific data. Record the version actually serving each observation so a dashboard does not mix old and new responses into one reassuring average.

Define a minimum sample and observation duration before seeing the results. Zero observed errors in ten requests is weak evidence about a rare failure. If requests are independent with a fixed error probability, a binomial model connects sample size to possible observations. Real failures may cluster by user or payload, so inspect those slices too. Missing telemetry should block or explicitly qualify a promotion decision instead of silently being interpreted as zero failures.

Make abort behavior observable and rehearsed

A rolling Kubernetes Deployment manages replica replacement and availability constraints, but those mechanics do not by themselves establish application correctness. A progressive-delivery policy adds the team's decision rules around behavior. Decide whether each breach pauses for investigation or automatically returns traffic to the stable version. Use deadlines on analysis steps so the fleet cannot remain indefinitely in an expensive or ambiguous transitional state without an owner.

Test the abort path with an intentionally failing release in a safe environment. Verify that traffic moves, new errors stop, old capacity is healthy, and any background jobs or schema changes are handled. Record detection time and recovery time separately. Promotion should be an auditable decision tied to artifact identity and evidence, not a green button whose meaning changes between teams. A rollout experiment is complete only when its steady-state and failure behavior are both understood.

E[K]=np,P(K=0)=(1−p)nE[K]=np,\qquad P(K=0)=(1-p)^n
K counts failed requests among n independent requests with a constant failure probability p. The expectation and zero-failure probability are teaching calculations, not a guarantee that production requests are independent or that one metric establishes release safety.↔ Scroll wide equations horizontally.

Size and observe a ten-percent canary

A service receives 1,200 requests/s. A routed canary gets 10%, with three replicas each measured at 60 requests/s. Observe for ten minutes. Compare expected failures at baseline rate 0.2% and a hypothetical regressed rate 0.6%.

  1. Canary arrival rate is 0.10 × 1,200 = 120 requests/s. Three replicas provide a measured total of 180 requests/s, so ideal utilization is 120 / 180 ≈ 66.7%.

  2. Ten minutes is 600 seconds. At a steady 120 requests/s, the canary receives 72,000 requests during the observation interval.

  3. Expected failures are 72,000 × 0.002 = 144 at baseline and 72,000 × 0.006 = 432 under the hypothetical regression, a difference of 288 expected failures.

  4. Those expectations are not a promotion test by themselves. Compare actual contemporaneous cohorts, confidence and clustering, latency, missing telemetry, and the predeclared abort criteria.

The result

The canary has modeled capacity headroom and a substantial sample, while the decision still requires comparable observations and explicit gates.

Explore the model

Interactive lab

Interactive lab · probability

Observe what a small sample can miss

Set p to a hypothetical request failure probability and n to independent observed requests. Exact binomial bars show possible failure counts, mean np, and variance np(1 − p). The panel does not sample traffic or perform a sequential canary significance test.

Observe what a small sample can missThese are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Current readings: Mean · np: 5; Variance · np(1−p): 2.5; Total probability: 1. All parameters are adjustable with the labeled controls below.03581000.070.150.220.3Successes kP(X = k)k = 0 · P = 9.77e-4
Exact probabilityHighlighted outcomeMean np
Mean · np
5
Variance · np(1−p)
2.5
Total probability
1
t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA quiet small sample can coexist with a meaningful underlying failure probability.

Check your understanding

A canary reports zero errors because its telemetry pipeline stopped. What should an evidence-based gate do?

Try it yourself

A 5% canary receives part of a 2,000-request/s service. It runs for five minutes. Calculate its request count and expected failures at p = 0.01. If each replica sustains 80 requests/s, is one canary replica sufficient under the simple capacity model?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Calculate traffic first, multiply by 300 seconds, and compare arrival rate with per-replica capacity.

Compare with the solution

Canary traffic is 100 requests/s. Five minutes produces 30,000 requests, with 300 expected failures at a 1% rate. One replica measured at 80 requests/s is insufficient for the 100-request/s arrival rate; at least two are required even before deciding on additional headroom. A precisely routed small traffic percentage can still overload an undersized canary.

Take it with you

  • Traffic allocation and replica capacity must agree.
  • Promotion requires comparable evidence and a defined observation window.
  • Abort includes capacity and compatibility, not only a routing change.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.