Test cross-shard invariants and recovery histories

For an internal transfer system, a conserved total and nonnegative per-account balances are useful invariants, provided fees, deposits, withdrawals, and in-flight representations are specified. Conservation alone is weak: a broken transfer that does nothing preserves the total. Record operation identity, input amounts, observed reads, invocation and response boundaries, and terminal evidence. A reference model should describe allowed operations independently of the storage implementation, so the checker is not simply repeating the same arithmetic and bugs through another code path.

Atomicity, isolation, and durability require different observations. All required participant outcomes should agree; read histories should satisfy the promised isolation model; acknowledged durable effects should survive failures within the declared storage model. A final-state scan cannot by itself establish serializability of every earlier read. Conversely, a serialization checker may accept an incomplete history only after considering possible outcomes of operations whose replies were lost. Do not remove ambiguous commits from the record merely because they make the oracle harder to write.

Choose crashes immediately before and after prepare persistence, decision persistence, participant application, and response delivery. Partition the coordinator from one participant while leaving other links available. Delay or duplicate messages where the transport model permits it, then restart processes with their intended durable state. Separately test storage-loss or Byzantine behavior only if the promised fault model includes it. Calling every fault a network outage hides which assumptions the protocol relies on and can lead to either false failures or unjustified confidence.

Use deterministic seeds or controlled scheduling barriers to reproduce critical interleavings. Preserve the seed, configuration, binary version, and event trace for a failing run. Reduce a long failing schedule while keeping the violation, so a reviewer can see which dependency or durability edge was missing. FoundationDB’s simulation work demonstrates the value of integrating fault exploration with system execution, but simulation remains bounded by the modeled environment. Passing simulated tests does not prove correctness for unmodeled device behavior, clock assumptions, or application misuse.

After healing faults, check that unresolved work reaches a valid outcome when the protocol’s liveness assumptions hold. An ordinary two-phase-commit participant may legitimately remain blocked while its decision authority is unavailable. A bounded test timeout detects failure to finish within that experiment; it does not automatically distinguish a liveness defect from violated recovery assumptions. Report safety violations separately from unresolved outcomes and slow progress. Keep sufficient evidence to explain whether locks, decision records, or compensation obligations prevent quiescence.

The sampling lab illustrates uncertainty in a deliberately artificial test process. Reset, set success probability to 0.1 and independent samples to 20, predict the sample-mean variance, then Step. For a separate comparison, Reset and explicitly set probability to 0.1 and samples to 100, predict, then Step. The exact binomial masses describe independent marked outcomes; playback highlights possibilities rather than running tests. A smaller variance in this model cannot prove a transaction invariant or establish that fault schedules in a real campaign are independent or representative.