Retries change the load that changes their failure probability
Model bounded attempt amplification, distinguish a constant-probability baseline from overload feedback, and connect retry budgets with deadlines and safe side effects.
On this page
The question for today
What you’ll understand
- Compute expected attempts under a bounded independent baseline.
- Explain how load-dependent failures can create positive retry feedback.
- Distinguish backoff, jitter, admission control, and idempotency guarantees.
A retry can rescue a transient failure while adding work to the same service that is already struggling. If extra attempts increase latency and timeouts, they can generate still more attempts. The service's failure probability is then part of a feedback loop rather than a fixed number. Start with a simple bounded-attempt calculation, then inspect precisely which assumptions fail as offered work approaches capacity.
Count attempts before treating retries as free reliability
Assume each failed attempt independently triggers one next attempt, with constant failure probability p and at most N total attempts. Expected attempts are 1+p+⋯+p^(N−1), because attempt j occurs only if all previous ones failed. This model ignores backoff scheduling, deadlines, cancellation, and load-dependent outcomes. It is useful for an initial capacity estimate, but successful delivery is not the same as exactly-once business effect. A timeout can follow a committed action, so every retried state-changing request still needs a compatible identity and effect contract.
Layered retries multiply worst-case work. If three layers each allow three attempts of their child operation, one original action can trigger up to 27 leaf attempts under the full nested retry path. A global retry budget or a single intentional retry layer makes that amplification easier to reason about. Include attempts already in flight when cancellation arrives; sending a cancel signal does not instantly reclaim their server CPU. Measure original operations, attempts, accepted work, and completed useful results separately so extra traffic is not mistaken for healthy demand growth.
A baseline can become inconsistent after its own amplification
Multiply original operation rate by expected attempts to estimate offered attempt rate under the fixed-probability baseline. Compare that rate with usable service capacity. When the estimate exceeds capacity, a stable low-delay interpretation is no longer credible without admission control, rejection, or changed behavior. Failure probability can rise with queueing and deadline misses, so recalculating with the old p is not self-consistent. The apparent retry improvement can therefore disappear precisely when many callers try it at once. Shared backends can propagate the overload to otherwise healthy callers.
An unlimited branching toy makes one instability condition visible: if every failed attempt produces m descendant retries on average, expected offspring is mp. The expected total family is finite only below one under the idealized independent branching assumptions. Real systems have attempt caps and deadlines, so this is not a literal claim that production generates infinitely many requests. Finite caps can still permit a damaging burst or sustain offered work above service. A feedback analysis must include arrival timing, resource retention, failure dependence, and what work continues after callers have given up.
Use controls that address different parts of the loop
Backoff delays attempts, jitter reduces synchronization, and retry caps bound attempts per operation. None creates service capacity or guarantees a safe side effect. Admission control, load shedding, bounded queues, and retry budgets can prevent low-value attempts from consuming the recovery headroom. Propagate end-to-end deadlines so a downstream retry does not restart the user's clock, and avoid retrying deterministic invalid input. Validate behavior during a recovery ramp: many queued clients becoming eligible together can recreate overload after the original fault has gone away.
The queue lab isolates offered-rate pressure without retry generation. For EACH run Reset, set BOTH rates, predict, then Play or Step: compare (λ,μ)=(4,3) and (2,3). Starting from backlog four at t=0, one grows and the other drains by four seconds. Edits retain t and recompute from four rather than a live policy change. There is no branching, backoff, deadline, cancellation, or load-dependent failure probability. Use the worked amplification calculation to choose a hypothetical offered rate, and use the queue only to reason about the resulting constant-rate imbalance.
Work it through
A changed failure rate can cross the capacity limit
A fictional service receives 70 original operations per second and can serve 100 attempts per second. Each operation allows three total attempts. Compare fixed independent failure baselines p=0.2 and p=0.5, ignoring all scheduling and deadlines.
At p=0.2, expected attempts are 1+0.2+0.04=1.24, giving 70×1.24=86.8 offered attempts per second.
At p=0.5, expected attempts are 1+0.5+0.25=1.75, giving 122.5 attempts per second, above the assumed capacity of 100.
The second baseline cannot describe a stable low-delay service without additional controls. Queue growth and deadline misses may change p further, so the arithmetic exposes a feedback risk rather than proving the observed system will settle at 122.5.
The same bounded retry policy can fit one hypothetical failure regime and overload another. Capacity and retry assumptions must be evaluated together.
The living notebook
Explore the model
Interactive lab · queue
Amplified offered work can exceed service
For EACH run Reset, set BOTH λ and μ, predict, then Play or Step. Compare (4,3) growth with (2,3) draining from four at t=0 to zero at t=4 seconds. Edits retain t/recompute from four. This constant-rate fluid model contains no retries, branching, backoff, jitter, deadlines, or load-dependent errors.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forRetries need a separate amplification and timing model; the queue only exposes the consequence of sustained offered load above service.
Pause & reason
Check your understanding
Your turn
Try it yourself
A policy permits four total attempts with independent fixed p=0.5. Calculate expected attempts and the offered rate for 40 original operations per second. What is the maximum leaf-attempt count if two nested layers each allow four attempts?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Expected amplification is a geometric sum; the nested worst-case count is a product, not that expectation.
Compare with the solution
Expected attempts are 1+0.5+0.25+0.125=1.875, giving 75 attempts per second. Two four-attempt layers can produce up to 16 leaf attempts on the full failure path. These are different quantities, and both omit timing, load dependence, and safe-effect semantics.
Keep the idea
Take it with you
- Bounded retries amplify offered work even under an independent baseline.
- Load-dependent failures can turn retry behavior into positive feedback.
- Backoff, admission, deadlines, and idempotency solve different requirements.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.