Randomize the stream you actually have

TensorFlow 2.16.1 Dataset.shuffle documents a streaming procedure that fills a buffer, randomly selects an element, and replaces the selected entry from the input while records remain. If the buffer contains only the first B records, the first output can only come from those records. Buffer size one preserves source order. A buffer covering a finite dataset permits a full shuffle under the algorithm's random-selection assumptions, but retaining every decoded example can be expensive. Shuffling a compact manifest of identities is a different memory choice from buffering full images or token arrays.

Examine the source ordering that the buffer sees: filenames, within-file records, interleaving, filtering, and partitioning all matter. Shuffling file order does not automatically shuffle records within each large file. A local worker shuffle cannot move a sample into another worker's shard, so it differs from a global permutation followed by partitioning. Increasing a buffer can improve mixing while leaving correlations at longer scales. Record the intended visitation policy and inspect class, source, and length runs over the actual batch stream rather than assuming a nonzero shuffle option guarantees every desirable distribution property.

A uniform permutation of a fixed finite dataset samples without replacement within an epoch. Independent Bernoulli calculations instead correspond to independent draws with a fixed class probability, such as replacement sampling under a stated model. Those are different distributions. When a rare class exists in a fixed count, use the finite-population model for a question about a uniformly chosen subset. When a bounded buffer cannot yet reach the class at all, even that global subset model is inappropriate for the early stream. State the available population and selection process before interpreting a probability.

Seeds determine random choices only together with the algorithm, dataset snapshot, epoch handling, and random-state evolution. TensorFlow's reshuffle_each_iteration controls whether iterations use a changing shuffle order; PyTorch samplers have their own seed and epoch contracts. Transform randomness and worker scheduling can introduce additional variation. Conversely, repeatedly resetting every worker to an identical seed can create correlated augmentation choices. Test both replay and distribution properties, because they answer different questions. A fixed seed is useful for diagnosis, not a certificate that a shuffle is statistically suitable or portable across framework versions.

Possible changes include increasing the buffer within a memory budget, shuffling manifest identities, interleaving source files, or deliberately stratifying samples. Each changes a specific part of the selection process, and stratification can intentionally change class weights. Check whether batching by length or asynchronous completion reintroduces order after shuffling. Compare per-step and per-epoch statistics as well as total counts; a class can have the correct epoch total while appearing in long runs that affect optimization. Preserve the exact policy in experiment metadata so a throughput improvement is not mistaken for the same training experiment.

The probability lab supplies an explicit independent comparison. Reset, set both p=0.2 and n=5, predict the zero-success bar, then inspect. It is 0.8⁵=0.32768 for five independent draws with the stated common probability. No random stream is generated, and the instrument has neither a shuffle buffer nor finite-population depletion. Compare this assumption with a real finite corpus or blocked source instead of treating the bar as a measured shuffle-quality score. The useful lesson is to choose a model that matches the visitation mechanism before interpreting a batch-composition number.