research28 min studyLesson 889 in Engineering

Runtime budgets: reserve bounded work before dispatching it

Separate usage estimates from enforceable reservations across tokens, tool attempts, compute time, retries, and concurrent children.

On this page

What you’ll understand

  • Compute remaining capacity from committed usage and outstanding reservations.
  • Reserve concurrent work atomically under explicit accounting units.
  • Settle actual usage without releasing uncertain in-flight obligations prematurely.

An estimated token count or expected tool cost is not a hard budget. Concurrent calls can each fit the apparent remaining allowance and collectively overspend it. A proposed enforcement design reserves a bounded amount before dispatch, settles measured use afterward, and retains uncertain obligations across timeout and recovery. Its guarantee depends on actual provider limits and complete accounting.

Define the resource vector and its measurement boundary

Track separate dimensions such as tokens, tool attempts, billed units, compute seconds, and wall-clock deadlines. They are not interchangeable without a declared conversion and price model. Count input tokens with the applicable tokenizer and include the prompt material actually sent. A generation output cap can bound one component only if the provider enforces it; hidden or separately billed work may require additional reserves or a softer policy. Label uncertain forecasts as estimates rather than pretending every vendor exposes a strict total-cost ceiling.

Tool retries, rejected attempts, deduplicated effects, cancellation, and cleanup can still consume resources. Decide whether a tool budget counts transport attempts, accepted executions, or logical effects, and record each where useful. A child run should receive a bounded allocation from its parent rather than independently treating the parent's whole budget as available. Lease expiry does not prove a remote request stopped, so reclaiming its reservation purely because a local timer fired can allow oversubscription.

A=B−U−R,U+R≤BA=B-U-R,\qquad U+R\leq B
B is a nonnegative resource-budget vector, U measured committed usage, R outstanding reserved upper bounds, and A available capacity, with inequalities componentwise. The desired invariant requires atomic ledger operations and true bounds on chargeable work; estimates alone cannot establish it.↔ Scroll wide equations horizontally.

Admission and settlement must be atomic and recoverable

Before dispatch, atomically reserve a request's declared resource vector if every dimension fits. Bind that reservation to logical run, child, step, attempt, and provider identity as needed by the accounting policy. Competing workers must not both approve against the same stale allowance. The budget transition and dispatch intent should share a supported durable protocol, with recovery deciding whether work was sent. A process-local counter or an unprotected read-then-subtract is not sufficient for a shared hard limit.

After confirmed completion, move actual usage into committed consumption and release the unused part of its reservation. Preserve the original reservation and settlement event so duplicate receipts do not charge or release twice. If actual cost exceeds the reservation, record the overrun and follow a defined containment policy; silently clipping the counter hides a broken bound. If the outcome is unknown, retain or transfer the obligation until authoritative usage or a safe upper-bound settlement is available. Cancellation is not a zero-cost receipt.

U′=U+c,R′=R−r,A′=A+(r−c)U'=U+c,\qquad R'=R-r,\qquad A'=A+(r-c)
This settlement applies to one previously reserved vector r and confirmed actual cost c with 0≤c≤r. Reservation r is removed exactly once while c becomes committed use. Unknown or excess usage needs another explicit policy; the formula does not justify releasing resources before work is settled.↔ Scroll wide equations horizontally.

Test simultaneous admission and uncertain completion

Run competing reservations that individually fit but jointly exceed a dimension, then verify only the allowed subset commits. Inject crashes after reservation, after dispatch, and before settlement. Test retries, duplicate usage receipts, child allocation returns, and cleanup spending. Report both estimates and measured use so systematic under-reservation becomes visible. Enforcement metrics should identify the exhausted dimension and remaining obligations rather than reduce everything to a generic too-many-steps error.

The function lab shows a declared synthetic cost family, not measured runtime usage. Reset, set a=1 and x=1, predict y=1, then Play or Step. Separately Reset, set a=1 and x=2, predict y=4, then Play or Step. In a hypothetical workload whose cost scales quadratically with input size, doubling size needs four cost units rather than two. The graph has no tokenizer, provider price, budget ledger, or upper-bound enforcement. Use it to check an explicit estimate's arithmetic, then validate the real bound and reservation protocol.

Reserve and settle a three-resource request

Use budget B=(1,000 tokens,10 attempts,100 compute-seconds), committed U=(400,3,20), and outstanding R=(300,2,30). A new bounded request reserves (250,1,10) and later confirms actual cost (180,1,8).

  1. Initial available capacity is B−U−R=(300,5,50), so the new reservation fits every dimension.

  2. After reserving it, available capacity becomes (50,4,40). Those units cannot simultaneously be promised to another child.

  3. Settlement releases seventy unused tokens and two compute-seconds. New committed use is (580,4,28), old outstanding reservations remain (300,2,30), and available capacity is (120,4,42).

The result

Reservation and actual usage are different ledger stages. The example preserves the componentwise budget only because the request's actual cost stays within its stated bound.

Explore the model

Interactive lab

Interactive lab · function

A cost estimate needs an explicit scaling model

Reset, set a=1 and x=1, predict y=1, then Play or Step. Separately Reset, set a=1 and x=2, predict y=4, then Play or Step. This fixed quadratic is a synthetic cost proxy, with no tokenizer, provider billing, reservation ledger, or measured compute.

A cost estimate needs an explicit scaling modelThis lab uses only y = ax². Playback traces the curve within the fixed integration interval from 0 to x; it does not estimate numerical error or change the chosen point. Current readings: y at chosen x: 0.608; Tangent slope · 2ax: 1.35; Signed area · ax³/3: 0.182. All parameters are adjustable with the labeled controls below.−2.4−1.201.22.4−9−4.504.59xy(x, ax²)
y = ax²TangentSigned integral
y at chosen x
0.608
Tangent slope · 2ax
1.35
Signed area · ax³/3
0.182
t = 0 s

This lab uses only y = ax². Playback traces the curve within the fixed integration interval from 0 to x; it does not estimate numerical error or change the chosen point. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forArithmetic on a forecast does not enforce a budget; admission needs a real bound and atomic reservation.

Check your understanding

Why should a timeout not automatically release all of a tool attempt's reserved budget?

Try it yourself

A tool-attempt budget is ten, with four committed attempts and three reserved. Two workers each request a reservation of two. Under atomic admission with no intervening settlement, can both fit?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

The shared available count is ten minus both existing ledger components.

Compare with the solution

Only three attempts are available. The first two-attempt reservation can commit, leaving one; the second cannot also reserve two. An unprotected read by both workers could incorrectly approve four new attempts and exceed the budget by one.

Take it with you

  • Estimates, upper bounds, reservations, and actual usage are distinct.
  • Concurrent budgets require atomic admission and exactly-once ledger settlement.
  • Unknown or cancelled work can retain chargeable obligations.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.