advanced25 min studyLesson 389 in Engineering

Deployment freezes and risk budgets

Use a defined reliability budget and release policy to pause routine exposure, preserve recovery paths, and make bounded exceptions auditable.

On this page

What you’ll understand

  • Calculate an event-based error budget with a declared window and denominator.
  • Distinguish forecast release impact from a guaranteed bound.
  • Specify a freeze scope, recovery exception, ownership, and exit criteria.

A deployment freeze is a control on selected changes during a period of constrained reliability or operational attention. It is useful only when the scope and escape paths are clear. An error budget can inform the decision, but it is an accounting policy tied to an actual service objective and population. It is not a universal currency that turns every proposed change into a safely spendable number.

Define the budget before counting what was spent

For an event-based availability objective, identify eligible events, the good-event definition, and the reporting window. The allowed bad events are the eligible count multiplied by one minus the target good fraction. A request-based budget differs from a time-based downtime allowance and can weight users differently. Preserve that distinction when discussing an incident. A denominator that quietly excludes failing requests or changes with instrumentation can make a depleted service appear healthy without improving any user outcome.

Subtract observed bad events to track remaining accounting room, including an explicit overspend when it goes negative. Historical accounting is different from forecasting future traffic or a release's incremental harm. A fixed-window policy can behave differently near a boundary than a rolling window. The organization should define which releases pause at which threshold, which corrective work continues, and how to handle incomplete telemetry. The policy should be understandable before an incident creates pressure to invent a more convenient interpretation.

Ballowed=N(1−S),Bremaining=Ballowed−EB_{\rm allowed}=N(1-S),\qquad B_{\rm remaining}=B_{\rm allowed}-E
N is counted eligible events in the defined window, S the target good-event fraction, and E observed bad events under the same definition. Both B values count events; remaining can be negative. This event-based accounting is not a time-based availability formula or a prediction of future harm.↔ Scroll wide equations horizontally.

Use risk estimates without calling them guarantees

A staged release can estimate additional bad events from exposure and a plausible increase in error probability. That estimate depends on comparable populations, the exposure forecast, and a valid outcome definition. Rare severe failures, correlated users, shared dependencies, and data corruption may not fit a simple independent-event cost. Some changes should have hard guardrails regardless of the average budget remaining. A proposal should state uncertainty and a maximum permitted exposure rather than claiming that an expected two thousand errors can never become more.

A freeze can target routine new artifact promotions while allowing rollback, a compatible emergency fix, or infrastructure work required to recover reliability. Define a scoped exception with an owner, artifact/configuration identity, reason, expiry, guardrails, and recorded decision. Argo CD sync windows provide scheduling controls and configured override behavior, but they do not automatically encode an organization's full incident policy. Test that an intended recovery action remains possible and that an exception cannot silently become a permanent global bypass.

E[ΔE]≈Nexposedmax⁡(0,pc−pb)\mathbb E[\Delta E]\approx N_{\rm exposed}\max(0,p_c-p_b)
N_exposed is eligible candidate exposure and p_c−p_b an assumed incremental bad-event probability for a comparable population. The result counts expected extra bad events. It is not a worst-case bound, a causal estimate without a valid design, or permission to ignore severity and correlated failures.↔ Scroll wide equations horizontally.

An exit condition should restore decision quality

Specify how the freeze ends: sufficient service recovery, reliable telemetry, completion of a corrective action, or another documented policy condition. A calendar timeout alone may restore routine delivery before the failure mechanism is understood. Conversely, an indefinite freeze can accumulate a large batch of changes that becomes harder to assess when released together. Keep changes reviewable, preserve test execution and artifact preparation where policy allows, and restart exposure in controlled increments. The purpose is to manage service risk while retaining a usable recovery system.

In the probability lab, Reset, set both p=0.1 and n=20, predict two bad events on average, then Play or Step. Separately Reset, set both p=0.01 and n=20, predict 0.2, then Play or Step. Exact binomial probabilities show that an expectation can be fractional even though observed counts are integers. These independent equal-probability trials do not model a service budget, correlated outages, or a freeze policy. Explain why expected impact should be accompanied by uncertainty and guardrails rather than treated as a guaranteed spending limit.

Account for observed consumption and a conditional proposal

An event-based window contains 100 million eligible requests with a 99.9 percent good-event target. Observed bad requests are 70,000. A proposal exposes one million comparable requests to an assumed incremental bad-event probability of 0.002.

  1. Allowed bad requests are 100,000,000×0.001=100,000. Subtract 70,000 observed bad requests to obtain 30,000 remaining in this fixed accounting window.

  2. The proposal's conditional expected extra bad requests are 1,000,000×0.002=2,000. Relative to the remaining thirty thousand, that is about 6.67 percent.

  3. The two-thousand figure is an expectation under the exposure and comparison assumptions, not a ceiling. A freeze policy can still reject the proposal because of uncertainty, severe-failure guardrails, recovery status, or a documented threshold.

The result

Budget accounting clarifies a decision without replacing the service's severity model, uncertainty treatment, and explicit release policy.

Explore the model

Interactive lab

Interactive lab · probability

An expectation is not a maximum loss

Reset, set both p=0.1 and n=20, predict two expected bad events, then Play or Step. Separately Reset, set both p=0.01 and n=20, predict 0.2, then Play or Step. This exact independent binomial count is not a service-budget policy or a model of correlated release failures.

An expectation is not a maximum lossThese are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Current readings: Mean · np: 5; Variance · np(1−p): 2.5; Total probability: 1. All parameters are adjustable with the labeled controls below.03581000.070.150.220.3Successes kP(X = k)k = 0 · P = 9.77e-4
Exact probabilityHighlighted outcomeMean np
Mean · np
5
Variance · np(1−p)
2.5
Total probability
1
t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forExpected harm can inform a release decision, while uncertainty, severity, and bounded exposure remain separate controls.

Check your understanding

A proposed release has expected incremental errors below the remaining error budget. Which conclusion is justified?

Try it yourself

A window has 1.5 million eligible events and a 99.9 percent good-event target. Observed bad events total 1,800. Calculate remaining budget and explain one useful freeze distinction.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Allowed bad events are 0.001 times the eligible count; do not hide a negative remainder.

Compare with the solution

Allowed bad events are 1,500 and remaining budget is 1,500−1,800=−300 events, an overspend of three hundred. A policy may pause routine feature exposure while retaining a documented path for compatible recovery changes. A blanket block that prevents its own remediation is different from a scoped freeze with tested exceptions.

Take it with you

  • Align objective, eligible events, outcome definition, and reporting window.
  • Expected release impact is conditional and does not bound severe harm.
  • Define freeze scope, recovery exceptions, ownership, expiry, and exit evidence.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.