Assign stable samples to explicit training owners

A useful identity combines a versioned dataset snapshot with a stable record key or an unambiguous location within an immutable object. A pathname or row number alone can change meaning when files are replaced or filtered. Distinguish the source record from a transformed training occurrence: two random crops may share a source identity while having different augmentation identities. Preserve those relationships for auditing and replay. A worker-local counter identifies only a place in that worker's execution; it cannot prove that another worker did not emit the same source record under another counter.

Choose one canonical ordered manifest for index-based sharding, including a specified listing order and filtering policy. For fixed worker count W, assigning canonical index i to residue i modulo W partitions indices without overlap when each owner processes precisely its assigned set. Equal record counts do not imply equal decode costs or token counts. Hash-based ownership is another design, but its hash, key encoding, and snapshot must be fixed; do not depend on a process-randomized language hash. Changing W generally changes ownership, so elastic resizing needs a migration or restart rule rather than an assumption that local cursors remain meaningful.

Distributed training ranks and loader subprocesses form different layers. A rank-level sampler can divide map-style indices while its local loader workers execute those requests. An iterable dataset often controls its own traversal and must explicitly combine rank and worker information to avoid repeating streams. PyTorch 2.9 documents that iterable dataset replicas need appropriate configuration; it does not automatically prove every custom iterator is correctly sharded. Validate ownership with small identifiable records before trusting an estimated loader length. Keep dataset construction and source listing consistent across participating ranks.

Tail behavior is part of the experiment. Some synchronous trainers require equal step counts, so a sampler may pad the index list with repeated samples or drop an incomplete tail. PyTorch 2.9 DistributedSampler exposes such a choice, and its epoch seed must be advanced with set_epoch when using its shuffled epoch traversal. Those policies differ from an uneven exact partition that visits every index once. Document whether repeated or omitted occurrences are permitted and how loss weighting and evaluation handle them. A constant batch count can conceal either policy, so record actual identities and multiplicities rather than infer coverage from the number of steps.

Check a short epoch by gathering sample identities and comparing observed multiplicities with the declared visitation policy. Include nondivisible dataset sizes, empty shards, multiple loader workers, and rank-count changes. Distinguish source records intentionally sampled with replacement from accidental duplicates caused by replicated iterators. Snapshot identity, ordering, partition policy, epoch, and relevant random state belong with restart metadata. Avoid silently relisting a changed object-store prefix after recovery, because a perfectly deterministic modulo rule over a different manifest is a different experiment.

The modular lab exposes a narrower ownership hazard. For each separate run, Reset, set both modulus m=4 and additive step a, predict the visited residues, then Play or Step. Step a=1 visits all four residues; a=2 visits only zero and two before repeating. This exact additive orbit is not a distributed sampler or a hash implementation. It illustrates why a stride through shard labels must be checked for complete coverage: a deterministic rule can repeat predictably while omitting valid owners. Real sample coverage still requires inspecting the actual manifest and emitted identities.