Container forensics: reconstruct the failing transition

Record container ID, bundle/configuration identity, runtime version, host PID, process start time, and the relevant namespace identifiers. A host PID can be reused after exit; a later process with the same number is not the original instance. Runtime created and running states also describe different points: setup can create a container environment before the user program executes. If exec never succeeded, searching only application logs misses the failure. Align runtime errors, hook outcomes, and actual process observations on a consistent timeline.

Observe from the intended namespace. A path visible to the host inspector can resolve differently for the target process, and a proc mount can expose a different PID view. Record root, current directory, mountinfo, and descriptors alongside the process credentials, mappings, capabilities, no_new_privs, and seccomp state. A namespace link identifies a kernel namespace object, not an entire container security policy. Two processes sharing one namespace can still differ in credentials, cgroups, open files, and the other namespaces.

An unlinked file can remain allocated while an open reference or mapping still uses it. A detached mount or namespace object can also remain referenced through descriptors or other holders. Therefore absence from one pathname listing does not prove resource reclamation, and a stopped application does not prove every runtime or sidecar released its handles. Inspect actual holders before attributing retained disk space to an OverlayFS leak. The type and lifetime of the reference matter more than the human-readable path that disappeared.

Preserve evidence before aggressive cleanup changes it. Collect descriptor targets and mount relationships with appropriate permission, then distinguish a deleted filename, a whiteout, a detached mount, and a still-live underlying object. A namespace file descriptor can pin an object, but does not make a PID namespace whose init died usable for new processes. Joining namespaces or rerunning hooks can itself alter state and is not a neutral observation. Prefer a reproducible fixture when a hypothesis requires a state-changing experiment.

For CPU problems, compare quota, ancestors, usage, throttling, runnable work, and pressure. For memory problems, correlate charged usage and local versus hierarchical events with the same workload instance. For denied syscalls, identify the attempted operation, actual errno or signal, credentials, filter, and LSM evidence where available. Conventional status 137 points toward SIGKILL-style termination, not uniquely OOM. Preserve a competing explanation, such as orchestrator escalation, until evidence distinguishes it. Changing several controls at once removes the ability to attribute the result.

The queue lab illustrates evidence-collection pressure rather than the container itself. Reset, explicitly set arrival four and service three, predict pending observation work growing, then Play or Step. For a separate capacity comparison Reset and explicitly set arrival two and service three. Rate edits retain elapsed time and recompute from four initial items. This unbounded fluid model has no finite trace buffer, dropped events, runtime phases, mounts, descriptors, or kernel causes. It motivates checking collector capacity without proving which event an actual investigation missed.