Memory controls: pressure, hard limits, and OOM evidence
Distinguish cgroup v2 memory.high and memory.max, hierarchical charging, reclaim, swap, and evidence for why a process was killed.
On this page
The question for today
What you’ll understand
- Explain why crossing memory.high is not an automatic OOM kill.
- Interpret memory.max and ancestor accounting without adding overlapping metrics.
- Diagnose memory pressure and kills using cgroup events and process evidence.
An inference worker can slow dramatically before it is killed, or receive an allocation failure without a clean narrative that it crossed exactly one byte beyond its advertised limit. Linux memory control includes charging, reclaim, throttling, and OOM decisions, not just a process RSS threshold. This lesson concerns the Linux 6.12 cgroup v2 memory controller with explicitly enabled controls. Kernel configuration, swap policy, allocation type, hierarchy, and runtime version affect details, so a diagnosis should preserve those settings alongside the observed events.
High pressure and a hard limit have different actions
memory.high applies pressure and throttling to reduce usage through reclaim when the boundary is exceeded. Crossing high does not itself invoke the OOM killer, and usage can exceed the boundary. An application can still be killed for another reason, including other memory pressure, so observing high is not an explanation for every later exit. Use memory pressure and event counters to distinguish a workload spending time in reclaim from one that simply needs more CPU or waits for I/O.
memory.max is the hard-limit mechanism: when the charged usage cannot be reduced sufficiently at that boundary, cgroup OOM handling can occur. Temporary overshoot and allocation-specific failure behavior exist; not every failed allocation invokes the OOM killer. Swap has separate controls and affects whether anonymous memory can move out of RAM. Raising a limit changes containment and capacity tradeoffs, but it does not repair a leak or guarantee that enough physical memory exists for all siblings and the host.
Charges belong to a hierarchy, not a sum of process RSS
memory.current includes the cgroup's charged memory and its descendants. A parent's total therefore already includes child totals; adding parent and child readings double-counts the hierarchy. Charged usage includes more than anonymous application payload, such as relevant page cache and kernel memory. Some detailed memory.stat fields overlap or refine broader categories, so adding every line is also wrong. Shared pages and charge ownership mean summing process RSS is not an equivalent measurement of cgroup usage.
A memory area can retain its existing charge ownership when a process migrates, rather than all historical pages moving immediately to the new cgroup. Ancestor limits can bind even when a child appears below its local limit. Observe the full path and the controller's documented event scope: memory.events is ordinarily hierarchical, while memory.events.local reports local events, subject to relevant mount behavior. A group OOM policy can treat a workload as a unit, but protected tasks and the chosen policy prevent a universal statement that every process must always be killed together.
An exit code is a clue rather than a complete cause
A conventional exit status of 137 often encodes termination by signal nine, SIGKILL. It does not identify who sent the signal. An operator, orchestrator escalation, cgroup OOM, or another kernel event can produce a superficially similar result. Correlate the same cgroup instance's memory events, kernel logs where available, process lifetime, and runtime state. A restarted workload with reset counters must not be mistaken for the earlier instance. The nearest dashboard spike is not enough to establish the causal path.
The function lab illustrates a workload's possible quadratic payload component. Reset, explicitly set a=1 and x=1, predict y=1, then Play or Step; Reset and explicitly set a=1 and x=2 for a separate prediction of four. Interpret x as normalized sequence length for one dense square allocation under a stated model. The graph does not simulate memory.high, reclaim, swap, hierarchical charges, or OOM selection. Quadrupled payload can motivate a budget check without proving that a particular process will be killed.
Work it through
Read a parent budget without double counting
In a stable illustrative charge ledger, a parent has 50 MiB charged directly, child A has 300 MiB, and child B has 250 MiB. The parent's memory.max is 768 MiB. Assume no other charged descendants or concurrent changes. Compute total and nominal headroom.
The parent total is 50+300+250=600 MiB. Its memory.current-style aggregate already includes both child amounts under this stipulated ledger.
Nominal headroom is 768−600=168 MiB, or 168×1,048,576=176,160,768 bytes. Adding the 600 MiB parent total to its 550 MiB of children would incorrectly count their charges twice.
The headroom is an observation, not reserved memory. A new allocation can face ancestor constraints, concurrent sibling charges, reclaim behavior, or other resource limits. If A is below its own local maximum, the parent can still be the constraining boundary.
The ledger describes 600 MiB total and 168 MiB nominal parent headroom. It does not establish an allocation or latency guarantee.
The living notebook
Explore the model
Interactive lab · function
A quadratic payload is only one part of the charge
Compare fresh Reset → explicitly set both a and x → predict → Play or Step runs at (1,1) and (1,2). y=ax² illustrates a dense square payload quadrupling only. It does not compute process RSS, page-cache charges, memory.high pressure, swap, memory.max enforcement, or OOM decisions.
- y at chosen x
- 0.608
- Tangent slope · 2ax
- 1.35
- Signed area · ax³/3
- 0.182
This lab uses only y = ax². Playback traces the curve within the fixed integration interval from 0 to x; it does not estimate numerical error or change the chosen point. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forModel the application's payload, then inspect actual hierarchical charged memory and pressure rather than predicting a kill from payload size alone.
Pause & reason
Check your understanding
Your turn
Try it yourself
A workload reports 800 MiB charged usage, memory.high of 768 MiB, and finite memory.max of 1024 MiB. Calculate the high overage and nominal max headroom. If it later exits with conventional status 137, what evidence is still needed?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Subtract each boundary separately, then identify what a SIGKILL-style status does not name.
Compare with the solution
Usage is 32 MiB above high and nominally 224 MiB below max. This can coincide with reclaim pressure; it does not prove an immediate cgroup OOM kill. Status 137 commonly encodes SIGKILL, but the cause requires correlated evidence from the same cgroup instance's events, runtime lifecycle, and available kernel or operator records. Check ancestors and distinguish local from hierarchical counters. Counter resets, process replacement, and other senders of SIGKILL can invalidate a conclusion drawn from the exit code alone.
Keep the idea
Take it with you
- memory.high pressure and memory.max containment have different semantics.
- Parent totals include descendants; RSS sums and detailed counters are not interchangeable.
- Diagnose kills through correlated events and lifecycle identity rather than an exit code alone.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.