Distributed debugging and determinism

Bitwise replay means reproducing exactly the same stored values under a defined environment. Numerical agreement within a tolerance is weaker but can be appropriate for parallel floating-point computation. Statistical reproducibility asks whether repeated experiments support the same performance or quality conclusion. Choose the level required by the bug: a mismatched collective order needs event traces, while a claimed validation improvement may need repeated training runs rather than identical weights.

Random seeds initialize particular pseudorandom generators; they do not control every source of nondeterminism. Data-loader worker initialization, sampler order, random augmentation, asynchronous execution, and nondeterministic kernels need separate treatment. Floating-point reductions can differ when worker topology or execution order changes. Record software and driver versions, device type, world size, rank mapping, and deterministic settings. A checkpoint containing weights alone does not capture the complete replay state.

Give each trace event a job identity, rank, logical step, microbatch, operation category, and local sequence number. Wall clocks may be skewed, so use causal relationships such as send-before-receive and collective participation, not a naive global timestamp sort. Log summaries or hashes of selected tensors at carefully chosen boundaries. Hash equality can locate identical bytes, but different hashes do not quantify numerical error or explain whether a difference is acceptable.

When a collective stalls, inspect earlier errors on every rank and compare intended collective sequences. A data-dependent branch that performs an extra synchronization on one worker can deadlock otherwise healthy code. Reproduce with fewer ranks and smaller inputs while retaining the failing structure. Add synchronization or debug instrumentation deliberately, because it changes timing and can hide races. Keep the original trace so the reduced case can be checked against the observed failure.

A useful capsule includes exact input identities, code revision, dependencies, configuration, model and optimizer state, scheduler state, gradient-scaler state where used, and relevant random-generator states. Distributed samplers and input cursors also matter when resuming in the middle of an epoch. Save enough metadata to detect a changed world size or shard assignment rather than silently replaying a different example sequence.

Deterministic modes may choose slower algorithms or reject an operation without a deterministic implementation. They do not establish identical behavior across arbitrary releases or hardware. Run the capsule twice in its stated environment, compare the earliest checkpoints, then vary one dimension at a time. Once the issue is understood, retain a small regression reproducer. Restore production settings and measure again so the fix is validated under the execution mode that actually serves users or trains models.