Memory-pressure diagnosis

Start with useful completions, latency, rejected requests, and failures over a specific time interval. Then identify the process group, container, NUMA node, and host involved. A container can hit a local limit while host-wide available memory looks healthy. A process can have a huge virtual mapping without occupying that much RAM. A large cache can be useful rather than waste. Label every metric with its scope and units before comparing it with another; inconsistent scopes can create an apparently impossible incident out of perfectly valid measurements.

Collect an initial accounting picture: resident and proportional process mappings where appropriate, anonymous versus file-backed memory, dirty/writeback state, relevant cgroup usage and events, swap occupancy, and system available-memory estimates. Keep these quantities distinct rather than adding overlapping categories. A snapshot identifies plausible owners, but not their growth rates or the time spent waiting. Capture a time series around a reproducible workload phase. Record recent code, batch-size, worker-count, and kernel-policy changes so that later explanations can be tested against the actual timeline.

Rising major-fault rates with page-loading I/O and latency spikes suggest a different mechanism from high minor faults during first-touch initialization. Heavy reclaim scans with a poor reclaim yield can indicate a shortage of eligible cold pages rather than a lack of scanning effort. Repeated refaults plus stalled tasks and reduced useful throughput support working-set churn. Large-allocation stalls with modest total occupancy can point toward fragmentation or constrained zones. None of these patterns is a universal proof; require a proposed explanation to account for the timing and the scope of several observations.

Pressure Stall Information reports time lost to resource contention rather than bytes occupied. For memory pressure, some tracks intervals when at least some tasks are stalled; full tracks simultaneous stalling of all non-idle tasks in the measured scope. Its cumulative totals can be differenced over a chosen interval, while recent averages use their own windows. Do not sum overlapping some and full percentages: full is a subset. Compare a group's pressure with its useful completions and relevant I/O pressure; memory reclaim can create storage work that appears in more than one resource view.

Choose a reversible intervention tied to the hypothesis. Reducing batch size tests peak live demand; reducing concurrent workers tests overlap; shortening pinned-buffer lifetime tests an unreclaimable pool; changing a huge-page policy in an isolated test can investigate compaction costs. Preserve the requested output and compare useful throughput, latency, and memory behavior together. A fix that lowers one counter by dropping work silently is not a performance improvement. Record the new configuration and a rollback point so that the result remains reproducible after the incident.

Use two reset queue-lab runs as a diagnostic exercise. Press Reset, explicitly set both arrival four and service three, predict one-job-per-second growth from four, then Play or Step. Reset again, set both arrival two and service three, predict drainage from four after four seconds, then Play or Step. Parameter edits alone retain elapsed time and recompute the initial-value model; this comparison is not a live rate switch preserving backlog. It isolates a proposed demand change, but equal jobs without byte accounting, an allocator, or telemetry cannot identify a real incident. For your workload, write expected evidence, a disconfirming observation, and a safe intervention that could prove the hypothesis wrong.