Make retries visible as training occurrences

A source record can legitimately appear in several epochs or in multiple draws of replacement sampling. Deduplicating solely by source identity would incorrectly remove those intended occurrences. Define an occurrence identity using snapshot, epoch or traversal identifier, sampler position, and any required augmentation view. A delivery attempt is another identity attached to an occurrence; retrying the same occurrence should not masquerade as a new independent draw. Preserve these distinctions in diagnostics so a repeated source can be explained as deliberate sampling, retry, or accidental overlap between workers.

Workers may fetch and transform ahead of the trainer. A completed read does not prove a batch was delivered, and a delivered batch does not prove its gradients contributed to a committed optimizer step. Failure after any of those boundaries leaves different uncertainty. Acknowledgments should have a specified meaning, and recovery must use state consistent with that meaning. A durable source offset that advances before the relevant consumer state is saved can create omission after restart. Conversely, replaying uncommitted input is not necessarily a duplicate model update if the model is restored to the matching earlier state.

Corrupt records, temporary read failures, and worker process failures are different causes. A bounded retry may be sensible for a transient read, while repeatedly decoding a permanently corrupt record may never succeed. Skipping that record can be an authorized policy, but it changes observed coverage and may bias training if errors correlate with source or class. Record skipped identities and reasons, enforce the declared tolerance, and expose exhausted retries. A loader that simply returns fewer samples can otherwise appear faster while silently changing the experiment.

Replacement workers must recover the correct ownership and random state rather than restart every iterator at its first record. Map-style and iterable pipelines expose different control points, and framework versions offer different restart support. PyTorch 2.9 documents iterable-replica duplication hazards; TorchData 0.11.0 describes explicit loader state hooks and worker-state aggregation, with its stated topology limitations. Neither fact permits assuming arbitrary external side effects or optimizer updates are exactly once. If a transform writes shared state, its replay safety needs its own design beyond reproducing a tensor.

Inject controlled failures before fetch, after transformation, after delivery, and around a checkpoint in a small identifiable experiment. Compare the intended occurrence stream, delivered attempts, and model-consumption record. Include worker replacement and topology changes, because a local cursor may not describe the same shard under a new world size. Preserve a reproducible diagnostic setup while recognizing that nondeterministic scheduling changes where the failure lands. Define what the experiment guarantees: perhaps exact replay under a fixed topology, or an explicitly bounded repetition policy, rather than an unsupported universal visitation claim.

The probability lab models only hypothetical retry frequency. Reset, set both p=0.1 and n=10, predict the zero-event bar, then inspect. Under ten independent equal-probability retry indicators, the chance of at least one is 1−0.9¹⁰≈0.65132. Actual worker failures can share storage or host causes and need not be independent. The instrument does not crash workers, determine delivery status, or deduplicate model updates. Use the conditional calculation to motivate observing retries, then use actual occurrence identities and commit-boundary evidence to diagnose their effects.