Load shedding and admission

A request rate is meaningful only relative to a workload distribution. Ten short cache hits and ten long generation requests can impose very different CPU, accelerator-memory, and connection costs. Measure resource service demand by request class and use a conservative operating envelope validated under realistic bursts. If a node has eight cores, a budget of six CPU-seconds per second leaves nominal headroom, but memory or storage may still bind first. Protect every resource whose exhaustion changes the service's behavior materially.

A token bucket allows a sustained rate with bounded bursts; a concurrency limiter bounds simultaneous in-flight work; a bounded queue limits waiting work. These controls address different failure mechanisms. A rate limit alone does not prevent concurrency from growing when service times lengthen. A concurrency limit without tenant accounting can let one slow workload occupy every slot. Apply inexpensive checks early, make reserved and shared capacity explicit, and define whether excess work is rejected, scheduled later, or placed in a durable queue.

Work should not wait indefinitely merely because a queue has space. Compare estimated waiting and processing time with the remaining end-to-end deadline, and decline work that cannot usefully finish. The estimates have uncertainty, so measure missed deadlines and adjust conservatively. A rough queue bound can start from a tolerated waiting interval times measured completion rate, then be tested against the actual service-time distribution. Queue length is not interchangeable with queue age, especially when request costs vary or scheduling prioritizes particular classes.

Separate critical operations from work that may be delayed, but prevent callers from assigning themselves unlimited priority. Derive criticality at a trusted boundary and propagate it consistently. Reserve recovery capacity for health checks, cancellation, control traffic, and administrative access; otherwise overload may disable the very mechanisms needed to repair it. Reject cheaply where possible, return an actionable overload signal, and coordinate retry behavior. An upstream caller that immediately resubmits every rejection can consume CPU and network resources without increasing useful work.

An adaptive limiter needs a stable signal and a controlled response. Raising a limit after one fast sample and lowering it after one slow sample can oscillate, especially with delayed metrics. Test smoothing, minimum observation windows, separate increase and decrease behavior, and hysteresis around operating thresholds. Do not tune only at steady load: exercise a burst, dependency slowdown, client retry wave, and return to normal. Record how quickly useful throughput recovers and whether tenants with short inexpensive work remain protected.

The queue lab exposes one basic conservation law: when arrivals exceed service capacity, fluid backlog grows at their difference until arrivals fall or capacity rises. It begins with four waiting units and clamps the backlog at zero. Its separate M/M/1 mean residence time assumes Poisson arrivals and exponential independent service, so it is not a production admission controller. Use the lab to calculate drain time, then explain which measured cost classes, fairness rules, and deadlines a real implementation must add.