Outlier ejection: remove a bad destination without losing the fleet

Passive detection sees failures generated by actual traffic, such as upstream error responses, connection failures, resets, or timeouts. Active health checking sends its own probes. Discovery membership says that an endpoint is advertised, while Kubernetes readiness reports another configured condition. These signals can disagree legitimately: a health URL may succeed while an authenticated application route fails, or a network partition can affect only one source zone. Record the request path and observer before labeling an endpoint globally unhealthy.

Envoy v1.40.0 distinguishes locally originated connection-related errors from externally originated upstream errors when configured to split those categories. A timeout at the proxy is different evidence from an application returning HTTP 500. Protocol handling also matters: a TCP proxy cannot infer arbitrary application error semantics it does not parse. A threshold on consecutive failures reacts differently from a windowed failure percentage, and low-volume hosts may not provide enough samples for a statistical detector. Specify minimum sample conditions and what resets a consecutive counter.

Ejection generally marks a host unavailable to a particular proxy's ordinary load-balancing set for a configured period. Other proxies may have different observations and decisions. A cluster can also be shared across routes, so excluding one endpoint can affect traffic whose own requests did not produce the failures. Envoy provides enforcement settings, maximum ejection controls, and interactions with active health checking. Its panic behavior can also change whether unhealthy hosts are selected. Do not present ejection as an unconditional network-wide ban that every route and proxy immediately obeys.

A policy limiting how many hosts may be ejected prevents one detector from casually eliminating the entire service, but it can also leave some unhealthy hosts selectable. The exact percentage rounding, minimum-host behavior, and recovery rules are implementation-specific. Evaluate endpoint capacities rather than counts alone: losing one large endpoint may remove more capacity than losing several small ones. Recovery should consider whether returning hosts can handle full traffic immediately, whether active probes exercise the failing path, and whether repeated ejections reveal a shared dependency rather than independent bad machines.

Test one failing endpoint, a zone-local connectivity failure, a shared upstream outage, and insufficient remaining capacity. Observe why a host was detected, whether enforcement actually ejected it, and how request outcomes changed afterward. A falling error count at the ejected host can simply mean it receives no traffic, not that its underlying failure recovered. Track successful completed work and latency across the surviving set. Include the possibility that aggressive ejection and client retries create a cascade by overwhelming otherwise healthy endpoints.

The probability lab illustrates sample count only. Reset, set p=0.10 and n=20, predict two failures on average and variance 1.8, then Play or Step. Separately Reset, set p=0.50 and n=20, predict ten failures on average, then Play or Step. These are exact independent binomial probabilities with a fixed stipulated failure rate. They do not run a consecutive-error detector, eject hosts, or model correlated outages. Explain why twenty quiet observations cannot prove that a rarely used authenticated route works, and why the observed failure category must remain explicit.