Dependency budgets and error budgets

Choose the population of eligible original operations and define what counts as good. The denominator must not silently switch to successful requests, individual retry attempts, or only the traffic that reached a healthy backend. A count-based budget is the allowed bad fraction times eligible volume for the specified window. A time-based objective uses a different denominator and should not be mixed into the same arithmetic without a defined translation. For quality-sensitive AI features, a fast syntactically valid response may still fail the useful-outcome definition.

Inspect important cohorts as well as the aggregate. A small tenant population can experience severe failure while the global percentage remains within target. Define supported request classes, excluded conditions, and how partial responses are scored before incidents occur. Budget policies should name decision owners and actions, such as pausing a risky rollout while restoring reliability. They should also acknowledge that security fixes or urgent corrective changes may need a separate controlled process. An exhausted budget is evidence for a decision, not an automatic mechanical substitute for judgment.

If a request requires several dependencies, each dependency's failure can make the operation bad. Without independence, the union bound says the total probability is no greater than the sum of those failure-event probabilities. An allocation whose sum fits the end-to-end allowance is therefore conservative for that modeled path, provided the events use the same operation population and include all relevant causes. Independent-product calculations can be more precise only when their assumptions are justified. Do not add overlapping incident counts and call the total distinct affected users.

An optional dependency used by only a fraction of operations should be weighted by that path's exposure and outcome semantics. For example, a recommendation outage may omit a panel without failing checkout; it can still consume a separately defined quality budget. Retries alter the relationship between dependency attempt failures and original-operation failures. Measure whether the user operation actually recovered before its deadline rather than charging every failed internal attempt directly to the product budget. Preserve dependency attribution as labels so overlapping causes remain analyzable.

A burn rate of ten means the observed bad fraction is ten times the allowed fraction; it does not mean ten percent of the monthly budget has already been consumed. Under constant eligible-operation rate and constant burn, a full window's budget would be consumed in the window duration divided by burn. Actual remaining time also depends on budget already spent, traffic changes, and the rolling-window definition. Use multiple observation windows so a short burst and a persistent slow burn can both lead to appropriate investigation without identical alert behavior.

The probability lab shows exact binomial counts under independent identical trials. Set a success probability and trial count to examine the mean and variance, then explain why correlated outages violate the model. A deployment defect can cause a long run of failures rather than independent coin flips, and rare failures may be too sparsely sampled to support a precise estimate. Use the lab to distinguish expected counts from guarantees. Operational burn decisions must retain time structure, cohort exposure, and uncertainty from the actual service.