Circuit breakers: separate a recovery state machine from resource bounds

In a closed state, calls are allowed and their classified outcomes update a failure detector. Crossing a defined threshold opens the breaker, causing selected calls to fail quickly instead of reaching the dependency. After a configured delay, a half-open state admits a bounded probe population. Success can close the breaker, while failure can reopen it. Define the observation window, error classes, timer origin, number of probes, and concurrent access rules. Without these choices, the three-state drawing is a vocabulary list rather than an executable protocol.

A successful probe must test the dependency behavior the guarded operation needs. A trivial health response may not establish that a costly authenticated transaction works. Concurrent callers need coordination so half-open does not accidentally admit thousands of probes. Reopening or closing can also affect requests already in flight; the protocol must state whether they continue and how their late results are counted. A breaker is usually local to a process, operation, or dependency key unless an explicit distributed mechanism coordinates it.

Envoy v1.40.0 documents limits per upstream cluster and priority for maximum connections, pending requests, outstanding requests, active retries, and connection pools. A pending request waits for a suitable ready pool connection, which differs from an already outstanding upstream request. HTTP/2 multiplexing means a connection count is not a request-concurrency count. These resource mechanisms do not require the exact closed/open/half-open failure-window protocol described above. Inspect overflow counters and the relevant resource rather than inferring the cause from a generic circuit-open label.

The limits are distributed across proxy instances rather than coordinated as one service-wide quota. Worker threads within a proxy share limits, but the documented implementation can have eventual-consistency races, and connection allocation has specific exceptions. Therefore a configured limit is not a universal exact hard ceiling on every live statistic. A fleet of twelve proxies each configured with a nominal fifty-request budget does not thereby enforce a fifty-request global budget. Scope, priority, cluster identity, and the number of callers determine the aggregate offered concurrency.

A fast rejection prevents some waiting and resource consumption at the protected boundary, but an upstream client can immediately retry and recreate the load. Coordinate retry budgets, deadlines, backpressure, and admission policy across layers. Classify whether a rejection is overload, policy denial, no eligible endpoint, or application failure so the caller does not retry every response identically. Monitor useful completions alongside rejection counts: a low queue can mean healthy service or simply that all work is being rejected.

The queue lab exposes pressure without simulating either breaker. Reset, set λ=4 and μ=3, predict the four-unit initial backlog grows by one unit per second, then Play or Step. Separately Reset, set λ=2 and μ=3, predict it drains in four seconds, then Play or Step. There is no capacity cap, closed/open/half-open state, or rejection control in the instrument. Use its uncapped growth to explain why a real bounded queue needs an explicit admission rule, then draw the chosen recovery-state schedule separately.