Prove where a training step waits for its input

Collect a short representative trace after separating startup, compilation, and cache warmup from steady execution. Mark batch acquisition, preprocessing, host-to-device transfer, model work, and synchronization with consistent timing semantics. Asynchronous device launches mean a fast host function can enqueue work that finishes much later, so host durations alone are insufficient. TensorFlow's input analysis guide uses IteratorGetNext-related events to inspect whether batch acquisition waits. Event names and profiler details depend on framework version and execution mode. Interpret them within that trace instead of assuming every long step has the same cause.

Inspect each rank in synchronous training. One rank's expensive record or slow read can delay peers at a collective, making their visible wait look unrelated to input. Conversely, a ready input queue with long device memory or communication operations points elsewhere. Measure queue emptiness, decode time, remote requests, transfer overlap, and per-rank variation alongside the device timeline. Sampling and tracing impose overhead and may perturb scheduling, so compare with a lower-overhead baseline. Keep identities or safe summaries sufficient to connect slow batches with length and source without logging private training content unnecessarily.

A bounded run using already prepared batches can test how much the input path contributes, provided shapes, dtypes, batch semantics, transfer path, and distributed execution remain comparable. Repeating one small cached batch may remove realistic memory pressure or change length distribution, so describe its limits. If the test accelerates, it supports investigating removed input work; it does not identify whether storage, decoding, collation, or transfer was responsible. Next change one stage or instrument its waits. If it does not accelerate, revisit communication, compute, memory, and measurement boundaries before adding loader capacity.

Report useful tokens or examples per second and the corresponding objective, not only allocated tokens or worker output. Skipping slow corrupt records, shortening sequences, relaxing output ordering, or freezing augmentation may improve a timing metric by changing the experiment. Validate sample multiplicities and distribution after a performance change. Compare representative cold and warm states when caches matter, and retain topology and configuration. A successful optimization should reduce the identified critical-path wait while satisfying the experiment's declared identity, visitation, masking, and restart contracts.

Even with ready input, an accelerator kernel can be limited by its own memory traffic rather than arithmetic throughput. A bandwidth ceiling inside the device is different from waiting for a remote training record. Communication, launch overhead, or small kernels can also leave arithmetic units underused. Use the relevant device counters and timeline to choose the next hypothesis, and avoid translating a hardware utilization gauge directly into a loader recommendation. The tf.data framework and service papers describe pipeline resource and placement tradeoffs, but their reported systems results are not performance predictions for an unrelated model or cluster.

The roofline lab supplies a separate device-side ceiling example. For each view, Reset, set both bandwidth B=100 GB/s and operational intensity I, predict, then inspect. At I=2 FLOP/byte the synthetic roof is 200 GFLOP/s; at I=20 it is capped at 1000 GFLOP/s. Playback traces the selected roof without measuring a kernel. The model has no input queue, storage system, or rank communication and cannot diagnose starvation. Its role is to show why low arithmetic throughput can arise from a device-memory limit even when training batches are already available.