advanced26 min studyLesson 418 in Engineering

A corrective action needs a mechanism and an observable invariant

Turn incident findings into bounded admission, cancellation, and verification controls with explicit scope, ownership, and regression oracles.

On this page

What you’ll understand

  • Connect a proposed action to the failure mechanism it should interrupt.
  • Calculate a fictional concurrency budget with fixed and per-request memory.
  • Define an oracle that catches recurrence and tests the intended boundary.

An action item is not complete merely because it has an owner and a due date. It needs a reason to change the failure mechanism and evidence that the change works within its scope. Training reminders, dashboards, retry limits, admission controls, and architectural changes act at different points in a system. Choose the point that interrupts the observed path, then define how a future test or operational check would detect its failure. A closed ticket without that evidence can leave the original condition intact.

Translate the explanation into a control boundary

If unbounded concurrent inference requests retain activations until a shared GPU exhausts memory, an admission limit must act before those allocations occur. A metric added after allocation may improve detection but does not enforce the bound. A timeout also fails to release memory if cancellation never reaches the worker. Map each proposed action to the exact boundary it changes: prevent admission, bound retained work, release state, reduce propagation, detect failure, or accelerate recovery. State what remains possible after the action.

For every control, identify scope and bypass paths. A limit of ten in each of three independent dispatchers is not a global limit of ten on one shared device. Work started by background jobs or old binaries can escape a newly added frontend check. Likewise, limiting request count does not prove a byte bound when requests have very different shapes. A useful review asks whether the mechanism's assumptions match the real resource and whether the implementation owns all relevant admission and release paths.

Mused≤Mbase+c m+MreserveM_{\rm used}\leq M_{\rm base}+c\,m+M_{\rm reserve}
This fictional upper-budget model uses bytes throughout: M_base is bounded fixed memory, c admitted concurrent requests, m a justified per-request upper bound, and M_reserve the bounded remaining category. It is valid only if these categories cover the allocation paths and do not hide unbounded work.↔ Scroll wide equations horizontally.

The regression oracle should observe the claimed property

A concurrency fix can assert that outstanding allocations remain within the scoped budget while requests are admitted, canceled, timed out, and retried. Test the old failure fixture as a positive control for the oracle, then verify the changed behavior and useful service. An error-message check alone can miss a leak that no longer emits the same message. Also test release on exceptional paths: a control that blocks overload but never returns permits can eventually make every request wait forever.

Record the action's owner, rollout target, verification artifact, and operating assumptions. Separate implementation completion from verified deployment, especially when old workers can remain active. Include a measurable follow-up for operational controls whose effect appears only under sustained conditions. Avoid treating every action as equally valuable because it fits a checklist; prioritize by the mechanism it interrupts and the evidence available to show improvement. The goal is a testable change in system behavior, not a polished list of generic lessons learned.

c≤⌊Mlimit−Mbase−Mreservem⌋c\leq\left\lfloor\frac{M_{\rm limit}-M_{\rm base}-M_{\rm reserve}}{m}\right\rfloor
For m>0 and a nonnegative numerator, this count bound follows from the stated fixed/per-request memory model. It is not a universal GPU capacity rule: dynamic shapes, shared workspaces, allocator behavior, and omitted paths can invalidate the assumed component bounds.↔ Scroll wide equations horizontally.

Preserve useful service while testing the control

A strict cap can prevent one failure while producing unacceptable queueing or unfairness. Define how excess work is rejected or delayed, what clients should do, and whether deadlines propagate. Verify correctness and legitimate throughput as well as the resource invariant. If a client retries immediately after rejection, the control can shift pressure elsewhere. Corrective design therefore includes interactions with callers and dependencies, rather than assuming one local bound settles the entire availability objective.

The queue lab is a fixed-rate work illustration, not an admission limiter. Reset, set both λ=3 and μ=2, predict growing backlog from four items, then Play or Step. Reset and set both λ=2,μ=3 for a separate comparison. Editing preserves elapsed time and recomputes from the initial four. No finite queue, memory allocation, fairness policy, or cancellation path is implemented. Use the visual to ask whether the real action changes admitted work, completed work, or neither, and verify that change with the actual oracle.

Derive a scoped fictional memory cap

A fictional device budget is 8 GiB, with a proven 2 GiB fixed category, 1 GiB reserved for bounded other work, and a supported request-shape bound of 0.5 GiB per admitted request. Assume one global admission boundary covers all allocations.

  1. Subtract fixed and reserved categories: 8 − 2 − 1 = 5 GiB remain for concurrent requests. The reserve is already part of the total budget, not additional capacity.

  2. Divide by 0.5 GiB per request and take the integer floor: 5/0.5 = 10 concurrent requests. Under the stated bounds, ten uses the five-GiB request category exactly.

  3. Verify a global ten-request limit before allocation and release permits on every completion or cancellation path. Three dispatchers each allowing ten would permit thirty requests and would not satisfy this shared-device argument.

The result

The model permits at most ten concurrent supported requests at the global boundary. Its correctness depends on the claimed memory bounds and complete admission/release coverage.

Explore the model

Interactive lab

Interactive lab · queue

An action should change a specific work boundary

For each separate case, Reset, set both λ and μ, predict the four-item backlog trend, then Play or Step. This fluid model has no finite admission cap, memory accounting, cancellation, or fairness oracle.

An action should change a specific work boundaryFluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Current readings: Fluid backlog: 4; Net rate · λ − μ: −1 /s; M/M/1 mean residence: 1 s. All parameters are adjustable with the labeled controls below.02.557.51001.534.56Elapsed time · sFluid backlogλ < μ
Fluid backlogCurrent time
Fluid backlog
4
Net rate · λ − μ
−1 /s
M/M/1 mean residence
1 s
t = 0 s

Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA corrective action is justified by a verified mechanism and scoped invariant, not by the existence of a completed task.

Check your understanding

Which is the strongest evidence that a proposed concurrency correction addresses the stated memory failure?

Try it yourself

A fictional 16 GiB budget has 4 GiB fixed use, 2 GiB bounded reserve, and 0.5 GiB per supported request. Calculate the model's cap, then name an oracle needed for cancellation correctness.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Subtract the fixed categories before dividing. A cancellation should not merely stop the caller from waiting; inspect the retained resource and permit lifecycle.

Compare with the solution

The request category is 16 − 4 − 2 = 10 GiB, allowing floor(10/0.5) = 20 concurrent requests under the complete model. A cancellation oracle should verify that worker allocations are released and the global permit returns within the specified lifecycle, while no canceled work continues allocating outside the bound. A client-side timeout alone does not prove those backend effects.

Take it with you

  • Corrective actions should name the failure boundary they change.
  • Resource limits need justified component bounds and complete scope coverage.
  • Regression oracles must test mechanism behavior and useful service, including exceptional paths.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.