advanced25 min studyLesson 369 in Engineering

Reservations and capacity borrowing

Define entitlements, unused quota, revocation, and physical placement instead of treating borrowed resources as permanent capacity.

On this page

What you’ll understand

  • Separate a reservation, nominal quota, and opportunistic borrowing.
  • Calculate borrowing limits and reclaim amounts in one resource dimension.
  • Explain why revocation timing and physical capacity constrain an entitlement promise.

A research team owns an entitlement to part of a shared GPU pool but is idle overnight. Letting another team borrow that unused capacity can improve useful work. The difficult moment arrives when the owner returns: the platform must decide whether and how to reclaim resources, and borrowers need a contract for interruption. A quota number alone does not specify either the physical placement or the time within which the owner can start.

Define the entitlement and its scope

A nominal quota is an accounting entitlement under a particular queue policy. A time-bounded resource reservation can instead identify resources and an interval for selected users or jobs. Neither should be confused with a completed pod placement or an application-ready guarantee. Resource flavors, topology domains, and other dimensions still constrain what can run. State whether a promise concerns a quantity of compatible resources, particular nodes, or an admission priority when those resources become available.

Borrowing uses another eligible queue’s unused share under configured rules. Limits on borrowing and lending can restrict how much is exposed. The borrowed amount remains opportunistic unless the policy explicitly converts it to another entitlement. A tenant should know whether long non-checkpointable jobs are allowed to consume it and how much notice accompanies reclamation. Avoid selling the same nominal quantity as an unconditional immediate-start promise to two tenants when the physical resources can serve only one.

uA≤qA+min⁡(bA,ℓB,max⁡(0,qB−uB))u_A\leq q_A+\min(b_A,\ell_B,\max(0,q_B-u_B))
For one resource and two queues in this simplified model, A’s usage u_A is bounded by its nominal quota q_A plus its borrowing limit b_A, B’s lending limit ell_B, and B’s currently unused nominal quota. Units match throughout. Real cohort rules and other constraints can be richer.↔ Scroll wide equations horizontally.

Reclamation is a workflow, not arithmetic alone

When the owner submits work, the policy may prioritize its new admission or evict borrowers to reclaim enough quota. Eligibility and priority rules determine which work can be interrupted. The resource does not become physically free until victims exit and cleanup completes. A borrower may need to checkpoint, transfer state, or restart later. Track intent to reclaim, eviction, resource release, quota update, and the owner’s actual useful start as separate observations; otherwise a successful accounting decision can hide a long service delay.

Kueue separates job admission and quota management from Kubernetes pod-to-node scheduling. This distinction prevents an incorrect assumption that sufficient borrowed quota automatically implies a compatible node assignment. Also verify reclaim semantics for the installed policy rather than assuming every queue can preempt every other queue. A protected workload or topology mismatch may delay reclamation or require another candidate set. The owner’s entitlement must be explained within these conditions and the availability of the underlying cluster.

r=max⁡(0,uA−qA)r=\max(0,u_A-q_A)
If policy requires A to return entirely to its nominal quota q_A, usage u_A exceeds that entitlement by r resource units. This is an amount to reclaim, not a choice of indivisible victim jobs or a bound on reclamation latency.↔ Scroll wide equations horizontally.

Reserve enough certainty for the promised service

Time-bounded reservations can help plan large training runs or maintenance, but backfilling around them depends on runtime estimates and enforcement. A borrower whose execution cannot end in time can conflict with the reserved interval. Define what happens on overruns, failures, or changed reservations. Keep physical inventory and accounting synchronized during node replacement and autoscaling. Test a simultaneous return of several nominal owners rather than evaluating only the easy case where one owner reclaims a small amount from idle borrowers.

The queue lab illustrates the drain component only. Reset, set arrival rate to 0 and service rate to 4, predict when the initial four-unit backlog empties, then Step twice. For a separate run, Reset and explicitly set arrival rate to 2 and service rate to 4, predict the one-second count, then Step twice. These are independent fluid runs from four units. There are no tenants, quotas, indivisible jobs, checkpoint delays, or physical reservations, so an empty backlog here cannot certify a real borrowing-revocation promise.

Borrow within three different limits

Queue A has nominal quota 6 GPUs and borrowing limit 4. Queue B has nominal quota 10, currently uses 7, and may lend at most 2. Assume one compatible pool and the toy two-queue rule. A currently uses 6.

  1. B’s unused nominal amount is 10 − 7 = 3 GPUs. The allowed loan is the minimum of A’s limit 4, B’s lending limit 2, and unused amount 3: two GPUs.

  2. A can reach 6 + 2 = 8 GPUs in this scenario, not ten. The remaining unused nominal GPU at B is outside its stated lending allowance.

  3. If policy later requires A to return to its nominal quota, the reclaim amount is 8 − 6 = 2 GPUs. Selecting victims and waiting for actual release require the documented workflow beyond this arithmetic.

The result

Borrowable capacity is constrained by both sides’ policy and current unused entitlement. It is not an extra permanent two-GPU installation.

Explore the model

Interactive lab

Interactive lab · queue

Revocation includes an actual drain

Reset, set arrival rate λ = 0 and service rate μ = 4, predict drain from four units, then Step twice. Compare a fresh Reset with both λ = 2 and μ = 4, predicting the one-second count before two Steps. This fluid analogy has no quota, borrowers, or checkpointing.

Revocation includes an actual drainFluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Current readings: Fluid backlog: 4; Net rate · λ − μ: −1 /s; M/M/1 mean residence: 1 s. All parameters are adjustable with the labeled controls below.02.557.51001.534.56Elapsed time · sFluid backlogλ < μ
Fluid backlogCurrent time
Fluid backlog
4
Net rate · λ − μ
−1 /s
M/M/1 mean residence
1 s
t = 0 s

Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forAn amount to reclaim and the time to release real resources require different evidence.

Check your understanding

A borrowed quota has been revoked in accounting, but victim workers are still shutting down on the only compatible nodes. What is established?

Try it yourself

A has nominal quota 8, borrowing limit 5. B has nominal quota 12, uses 9, and lending limit 4. Under the toy rule, find A’s maximum usage. If A reaches that usage and must later return to nominal quota, how much is reclaimed?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Unused B quota can be tighter than either configured limit.

Compare with the solution

B has 12 − 9 = 3 unused units, so the loan is min(5,4,3) = 3. A can reach 11 units, and returning to nominal requires reclaiming 11 − 8 = 3. Indivisible jobs may force more than three units of eviction, and topology or protected-victim rules can prevent an immediate feasible reclaim path.

Take it with you

  • Nominal quota, physical reservation, and useful application start are different promises.
  • Borrowing is bounded by policy and unused compatible entitlement.
  • Reclamation needs victim eligibility, resource release, and an explicit timing contract.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.