Scale the stage that limits batch production

Express capacities in the same unit, such as completed batches per second with a fixed shape and composition. Reading records per second and decoding variable-length sequences per second are not directly comparable without the conversion. Measure representative data, including large images, long sequences, and expensive codecs. Averages can hide slow batches that stall synchronous ranks. Separate startup and cache warmup from a steady observation window, and include the collation work that assembles tensors. A worker's busy percentage alone does not identify whether it is decoding, waiting on storage, or contending for a shared resource.

With ideal overlap and no shared-resource interference, sustained pipeline throughput cannot exceed its slowest stage's capacity. Multiple independent decoders can increase one stage's capacity until another stage becomes limiting. In practice, readers and decoders may share memory bandwidth, cores, or network connections, so multiplying a single-worker rate is only an upper-bound starting point. Sweep a small number of configurations while observing delivered throughput, tail waits, CPU pressure, and memory. Keep the batch definition constant during comparison; changing sequence length or dropping expensive records changes the work being measured.

PyTorch 2.9 DataLoader can use subprocesses for dataset access, transforms, and collation. Native libraries may also use internal thread pools, so four workers each starting eight threads can compete for fewer available cores. Measure total runnable work and relevant library settings instead of equating processes with independent CPU capacity. Vectorizing an operation across a batch can reduce per-item scheduling overhead, while another operation may need per-example semantics. TensorFlow's performance guide illustrates parallel map, interleave, and vectorized transformations; these are mechanisms to evaluate for the actual workload, not guaranteed speedups for every input function.

Output ordering is another explicit choice. Waiting for an early slow task can cause a head-of-line stall even when later tasks are ready. Returning results as they finish may reduce that wait, but completion time can correlate with class or sequence length. PyTorch 2.9 warns that in_order=False can affect reproducibility and skew delivered data for imbalanced cases. Decide whether exact order is required, and inspect distribution changes if relaxing it. A throughput comparison must not quietly replace ordered epoch traversal with a schedule-dependent subset of faster samples.

For randomized transforms, distinguish each sample occurrence's intended random identity from whichever worker happens to execute it. A deliberate stateless seed derived from experiment seed, epoch, sample identity, and augmentation index can reduce scheduling dependence when the algorithm supports that contract. Otherwise capture and seed each relevant library's state, not just the framework RNG. Reproducibility still depends on supported operations and versions. Define how corrupt records or decoder failures are reported and handled. Silently skipping difficult inputs can change the training population, especially when corruption or decode cost correlates with source or class.

The queue lab isolates the capacity arithmetic for pending preprocessing work. For each separate run, Reset, set both arrival rate λ=4 and service rate μ, predict, then Play or Step. At μ=2 the initial four-job backlog grows by two jobs per second; at μ=5 a fresh backlog drains in four seconds. This deterministic fluid model does not launch workers, decode records, or model ordered completion. Use it to explain why insufficient aggregate service cannot be fixed by a bigger waiting queue, then measure the real resource and ordering constraints before changing concurrency.