Tenant budgets: coordinate rate, bursts, and in-flight work

Derive the tenant key from a trusted identity mapping, and carry it through asynchronous queues and retries. A user-controlled key allows identity rotation to evade a budget. Decide what is being charged: raw requests, input tokens, output tokens, bytes, accelerator milliseconds, or a conservative work estimate. A single short embedding and a long generation are not equal physical demands simply because both are HTTP requests. If the eventual cost is unknown at admission, reserve a bounded estimate and settle against actual consumption without allowing unbounded speculative work.

Charge retries and fan-out according to the resource obligation, not just the first public request. A gateway can admit one request that launches many retrieval calls or model attempts. If those downstream operations bypass the tenant budget, the upstream limiter measures an incomplete work path. Document who owns the reservation and when it is released or adjusted. Include denied work, estimation errors, and settlement failures in observability so that an attractive admitted-request counter cannot hide a growing pool of uncharged execution.

Giving every replica the full global rate and burst multiplies the allowance. One design uses a common decision service; another partitions the global budget into bounded local allocations. Leased allocations can reduce coordination frequency, but their ownership, expiry, replacement, and failover must prevent duplicate spend. A replacement replica must not reuse capacity that an old owner can still consume. Epochs or fencing can help enforce an ownership protocol, but naming a lease alone does not prove it. Define the failure policy when the coordination service is unavailable.

Static partitioning has a clear sum bound but can waste capacity when demand is uneven: one replica exhausts its share while another’s share sits idle. More dynamic borrowing can improve utilization at the cost of a more complex correctness argument. A local limiter can still protect a process from bursts before a global decision, as Envoy’s architecture describes. The two layers answer different questions. Test skewed routing, replica churn, partial coordinator failure, retries, and recovery rather than validate only the balanced all-replicas-healthy case.

Rate limits constrain admitted work over time; concurrency limits constrain simultaneously charged work. A slow downstream can create many in-flight operations even with a modest arrival rate. Cancellation is especially subtle: returning an HTTP error does not necessarily stop already launched GPU kernels or remote calls. Keep the relevant execution reservation until completion or a verified cancellation boundary, and distinguish a waiting-client permit from a physical-work permit. Otherwise clients can repeatedly cancel and relaunch while the reported concurrency remains low and real resource use grows.

Use separate fixed-rate queue runs as an overload analogy. Reset, explicitly set arrival two and service three, predict drainage from four initial items, then Play or Step. For a second run Reset and explicitly set arrival four and service three before predicting growth. Rate edits retain elapsed time and recompute the initial-value formula, rather than preserving a live backlog. The model has no token buckets, distributed leases, request costs, cancellation, or in-flight identities. Those accounting invariants require trace-based and failure-injection tests in the actual limiter.