Crash experiments that test a stated recovery invariant

Start with an application invariant, such as a public manifest naming a generation whose payload digest matches. Then define the acknowledgment promise: after durable success, recovery must expose that generation or a later valid committed generation according to the protocol. Before acknowledgment, an old or new complete generation may be allowed, while a manifest paired with partial payload is forbidden. This allowed-state set is the oracle. A filesystem that mounts cleanly can still fail it; a leftover temporary file may be harmless if the protocol explicitly permits and safely cleans it.

Keep the acknowledgment record outside the failure domain being tested, or use a harness whose observation semantics are explicit. Otherwise the test can lose the evidence telling it which operations were promised durable. Record operation identifiers, generation numbers, byte counts, synchronization results, and crash cut points. A wall-clock timestamp alone does not reconstruct persistence order or prove a syscall completed before the failure. Treat failed or interrupted operations as uncertain according to their contract rather than automatically classifying them as absent.

Killing only an application leaves the kernel, page cache, and device running; pending writeback can continue. It tests application recovery from process interruption but is not a power-loss test. Crashing a virtual machine can discard guest memory while host caching remains, depending on the virtual-disk setup. A physical power interruption has different behavior again, including device caches and power-loss protection. State precisely which buffers and components survive each experiment. Do not label every abrupt stop as a disk crash and then generalize its passing result to stronger faults.

A controlled block-device or virtual-storage fault-injection harness can explore selected reorderings, lost writes, and errors, but its model is also a specification with limits. Identify whether it represents flush and FUA correctly, whether sectors can tear, and which completed writes are eligible to disappear. Filesystem version, mount settings, kernel, virtual-disk cache configuration, device firmware, and initial layout belong with the result. Original crash-consistency research shows why application protocols need testing across persistence behaviors rather than assuming one filesystem's incidental ordering is portable.

On each run, restore the known starting image, execute one recorded operation trace, interrupt at the selected point, recover through the documented filesystem procedure, and inspect the application oracle before making repairs. Preserve the failed image, trace, configuration, and checksums. Reduce a failing sequence by removing unrelated operations while retaining the failure and prerequisite state. A small counterexample that names the broken ordering is more useful than a large collection of unexplained corrupt images. Rerun the proposed fix against the original failure and neighboring cuts.

The probability lab illustrates repeated independent detection opportunities, not exhaustive correctness. Let p be a hypothetical probability that one trial exposes a particular bug, and n the number of independent trials up to the lab's thirty-trial limit. Increase n and predict more probability of at least one detection; the zero-success bar gives the complement. Real cut points are often correlated and p is unknown, so a hundred clean runs cannot be converted into proof by inventing independence. The lab runs no filesystem and injects no crash; use it to explain why both systematic cases and repeated trials matter.