greatitself.
Continue
advanced28 min studyLesson 717 in Engineering

Choose a service unit before claiming tenant fairness

Account for variable prompt/decode costs, weighted service, bounded quanta, and backlog eligibility instead of equating equal request counts with fair resource shares.

Before you begin
On this page

The question for today

What you’ll understand

  • Define a scoped service-accounting function for heterogeneous generation work.
  • Calculate weighted budget shares and distinguish them from request shares.
  • Explain the assumptions needed for a fairness bound or virtual-service policy.

Equal request counts can be an unfair allocation when one tenant submits short prompts and another submits long generations. Equal token counts can also hide differences between prompt and decode cost or between short and long contexts. Fairness begins by choosing the service measure and the eligible population, then specifying weights and how idle tenants are handled. A scheduler should report those choices with its latency results so that a high-throughput aggregate cannot conceal one tenant's starvation.

Define what the policy charges as service

A simple accounting function can charge prompt tokens and output tokens at different declared rates. A more detailed function can use profiled work or context-sensitive terms. These are policy units: they need not equal an exact decomposition of simultaneous GPU time among requests in a batch. Calibrate the measure against the resource and product goal, and say what it omits. Charging only completed requests can miss service already consumed by long, canceled, or ultimately failed generations.

The original Virtual Token Counter work discusses configurable service functions and unknown output lengths in LLM serving. Its guarantees depend on its algorithm and assumptions; merely adding a counter to another scheduler does not inherit them. Count observed increments as work proceeds, reconcile estimates with realized work, and define how a new or returning tenant's counter is initialized. Otherwise a tenant can gain an unintended advantage by repeatedly appearing idle or by having its long work charged only after completion.

ci=αPi+βOi,α,β>0c_i=\alpha P_i+\beta O_i,\qquad \alpha,\beta>0
P_i and O_i are counted prompt and committed output tokens under the declared policy, with coefficients in service units per token. c_i has service units. This illustrative linear charge is not asserted to equal physical GPU time, especially under shared batching, prefix reuse, or context-dependent attention.↔ Scroll wide equations horizontally.

Weights allocate service among eligible backlogs

For continuously backlogged tenants and a divisible ideal service budget, weights determine proportions of the chosen service measure. They do not generally determine equal request counts or identical latency. A tenant with costly requests may complete fewer of them while receiving its fair service share. If a tenant becomes idle, state whether unused share is borrowed and how reentry affects accumulated credit. Memory constraints can temporarily make work ineligible even when its tenant has service entitlement.

A virtual-service approach can increment a tenant's counter by charged work divided by its weight and favor eligible tenants with less normalized service. A complete algorithm also needs tie-breaking, arrival handling, bounded work increments or another analysis, and interaction with prompt admission and ongoing decode. Hard deadlines and fairness can conflict under overload; the service contract must choose how to resolve that conflict instead of promising both without sufficient capacity. Measure service lag and starvation windows as well as completed-request counts.

Bi=Bwi∑j∈Ewj,Δvi=ΔciwiB_i=B\frac{w_i}{\sum_{j\in E}w_j},\qquad \Delta v_i=\frac{\Delta c_i}{w_i}
B is a divisible service budget, E the explicitly eligible backlogged tenant set, and w_i>0 weights. B_i is an ideal share in service units; v increments are normalized service units. These equations do not prove a finite scheduler's fairness bound without its transition, quantum, and eligibility assumptions.↔ Scroll wide equations horizontally.

Test fairness with costs that differ on purpose

Create at least one workload where equal request counts imply unequal charged service. Add a bursty returning tenant, one tenant with only long outputs, and a tenant constrained by memory or cancellation. Compare the offered demand, eligible intervals, realized charged service, wait times, and goodput by tenant. A system can appear fair on aggregate completed counts while a slow class accumulates indefinitely. Keep policy changes separate from changing the offered mix so the comparison identifies what the scheduler actually improved.

The modular lab shows only an equal-cost slot pattern. Reset, set both m=4,a=1, assign residue zero to A and residues one through three to B, then predict one A slot and three B slots over four Steps. Reset and set both m=2,a=1 for equal slots. These are paper labels on a deterministic orbit, not a tenant scheduler. Every labeled slot must represent an equal service quantum for the ratio to mean service fairness; assigning one variable-cost request to each slot would invalidate that interpretation.

Work it through

Allocate weighted service rather than equal request counts

Two continuously backlogged eligible tenants A and B have weights one and three. A divisible interval offers 120 service units. A's requests each cost two units and B's ten. Assume no overhead, memory exclusions, or partial-request accounting complications.

  1. The ideal weight shares are A:120×1/4=30 units and B:120×3/4=90 units. The policy is defined in service units, not request counts.

  2. Those budgets complete fifteen A requests and nine B requests because 30/2=15 and 90/10=9. Different completion counts are consistent with the declared weighted service allocation.

  3. Equal counts would produce a different service ratio: one A request costs two while one B request costs ten. Evaluate fairness against the chosen service measure and weights rather than declaring fifteen versus nine intrinsically unfair.

The result

The ideal allocation is thirty and ninety service units, corresponding to fifteen and nine requests in this exact fixture. Real discrete scheduling needs its own lag and eligibility analysis.

The living notebook

Explore the model

Interactive lab

Interactive lab · modular

Label equal-cost slots before interpreting their ratio

Reset → set both m=4,a=1; label residue 0 as A and 1–3 as B, then take four Steps for one A and three B visits. Reset → set both m=2,a=1 for equal visits. The orbit has no requests or tenant scheduler; fairness interpretation requires equal-cost service quanta.

Label equal-cost slots before interpreting their ratioAddition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Current readings: Residue r0: 0; Orbit period · m/gcd(m,a): 12; Distinct residues visited: 1 of 12. All parameters are adjustable with the labeled controls below.Repeated addition modulo 12: the current residue is 0, and the orbit period is 12.01234567891011+ 5mod 12An additive orbitn = 0rₙ = 0Period = 12Cycle starts at 0:0 → 5 → 10 → 3 →8 → 1 → 6 → 11 →4 → 9 → 2 → 7 → 0Exact residues · two additions per second
Additive orbitCurrent residue / next edge
Residue r0
0
Orbit period · m/gcd(m,a)
12
Distinct residues visited
1 of 12
t = 0 s

Addition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA ratio of slots or requests becomes a service share only under an explicit cost convention.

Pause & reason

Check your understanding

A receives fifteen short requests and B receives nine expensive requests under weights one and three. Which evidence determines whether the service allocation follows the policy?

Your turn

Try it yourself

Under c=αP+βO with α=0.01 and β=0.1 service units per token, compare request A with P=400,O=20 and request B with P=100,O=100. Do equal request counts provide equal charged service?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Compute prompt and decode charges separately, then add within each request.

Compare with the solution

A costs 0.01×400+0.1×20=4+2=6 units. B costs 0.01×100+0.1×100=1+10=11 units. Equal request counts therefore allocate different charged service, with B receiving 11/6 times as much per request under this model. The coefficients are a declared accounting policy, not measured physical time; a deployment should justify them or use another explicit service function.

Keep the idea

Take it with you

  • Fairness requires an explicit cost measure, weights, and eligibility policy.
  • Prompt/output counts and completed-request counts can represent very different service amounts.
  • Algorithmic fairness bounds require the actual scheduler assumptions and should be tested per tenant.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.