Load balancing: select using a signal with a known scope

A TCP-level balancer often selects an endpoint when a connection is created. If that connection then carries many HTTP/2 or gRPC streams, its traffic can remain concentrated on one backend. An HTTP-aware proxy can select an upstream for individual requests or streams, subject to its routing and pooling behavior. A long-lived streaming request still occupies its chosen endpoint rather than being transparently redistributed every time load changes. Count new connections, active streams, and requests separately when diagnosing an apparently uneven service.

Round robin rotates choices among available endpoints, potentially with weights, but equal assignment counts do not imply equal execution cost. One endpoint may receive expensive requests or have less CPU capacity. Least-request policies use active work counts as a signal; Envoy v1.40.0 documents different selection behavior for equal versus unequal endpoint weights. Latency-aware methods such as Linkerd 2.18's documented EWMA approach use another observation boundary. Neither an active count nor a recent response time is a complete instantaneous measurement of backend resource saturation.

A power-of-two-choices design samples a small candidate set and chooses the less loaded candidate according to its signal. This can avoid the coordination and herding of every caller choosing a globally minimum reported endpoint. However, each proxy generally sees only its own active requests unless an explicit reporting mechanism supplies broader information. Signals can be delayed or biased by successful completions. State the sampling and tie rules before deriving probabilities; two independent samples with replacement are not equivalent to selecting two distinct hosts.

Consistent hashing tries to keep a routing key associated with a relatively stable endpoint set. That can help cache affinity or stateful placement, but a popular key can overload its assigned endpoint. Hash balance across key space does not imply balance across request popularity or execution cost. Adding or removing endpoints changes some assignments, and locality or health policy can modify the effective set. Choose affinity because the application needs it, then measure skew and failure behavior rather than calling it a universal load-equalization mechanism.

Measure per-endpoint request starts, active work, completions, error rates, and resource use under comparable request mixes. A server receiving fewer requests can still consume more CPU if its requests are larger. A recovering endpoint may need a gradual ramp rather than its full nominal weight immediately. Test a slow endpoint, heterogeneous capacity, long streams, a hot affinity key, and a rolling membership change. Verify which policy layer selects locality, priority, subset, and final endpoint so that changing one weight is not mistaken for controlling the entire path.

The probability lab represents the explicitly hypothetical independent candidate draw. Reset, set p=0.20 and n=2, predict the bar for two high-load candidates has probability 0.04, then Play or Step. Separately Reset, set p=0.50 and n=2, predict that bar is 0.25, then Play or Step. The bars are exact binomial probabilities and do not implement a balancer or simulate response times. Explain why the candidate probability alone cannot establish that a selected server has spare CPU or that a latency percentile improves.