Reserve physical work as well as counting incoming requests

Apply limits using trusted tenant/account identity, with separate service-wide and dependency-specific budgets where needed. Per-IP counters can be useful for some abuse patterns but do not establish tenant identity or fair shared-account usage. Document burst and sustained limits, concurrent in-flight capacity, and window/refill semantics. A count limit should state whether retries, streaming requests, rejected requests and background tool work consume it. Otherwise a caller can shift expensive work onto an uncounted path without exceeding the advertised request rate.

Use the actual model/tokenizer and generation ceiling for reservations. A simple planning cost can weight prompt tokens, maximum generated tokens and bounded tool calls, but those coefficients must be declared and validated for the intended resource. A financial credit estimate is not a universal FLOP, latency or memory predictor. Physical occupancy can outlive a client connection, so also enforce the scheduler and worker limits required to keep the machine within capacity.

Concurrent workers must atomically compare-and-reserve against the shared budget. Separate local counters can all pass while the account total exceeds its cap. A retry can create new physical work even when a client reuses an identifier; only an implemented idempotency/result-reuse contract can prevent that duplication. Keep operation identity, reservation ownership and actual usage reconciliation linked so repeated completion events cannot refund twice or lose an unfinished job.

At completion, reconcile measured usage with the reservation under the declared billing and resource policy. A disconnected client or HTTP timeout does not prove the GPU, media decoder or external tool stopped. Release physical capacity only after the relevant work and buffers are actually quiescent or safely transferred to another accounted owner. Retain enough reservation for retries or uncertain external charges as specified; do not convert uncertain completion into unearned spend or capacity headroom.

Exercise two requests that each fit independently but exceed the shared cap together. Check the final reservation ledger and actual worker tasks. Include long output ceilings, many attachments, bounded tool fanout, retry races and cancellation during a stream. Verify both authorization and budget rejection on alternate paths. A service-wide upper limit can protect capacity while still producing unfair tenant allocation, so assess fairness and abuse policies separately from the hard resource invariant.

The queue lab makes a limited accepted-work imbalance visible. Reset, set both λ=3,μ=2, predict backlog eight at t=4, then take eight Steps. Reset and set both λ=2,μ=3 predicts zero at t=4 from the initial four. Parameter edits retain time and recompute from that initial state, so use separate fresh runs. This deterministic fluid model has no finite cap, token ledger, tenant identity, spend, cancellation or tail-latency distribution. It illustrates backlog growth, not an implemented limiter.