A training step ends when its longest required dependency path ends

Represent an operation or required wait as a node with a nonnegative duration. Directed edges indicate that one operation must finish before another can begin. Data dependencies, same-stream ordering, collective completion, and an optimizer's need for particular gradients all contribute edges. A CPU launch call is different from the device work it enqueues. Decide whether a step ends after launches, after device completion, or after a valid optimizer update; these boundaries answer different questions. Include input and communication only when the chosen end-to-end boundary actually requires them.

For an ideal dependency DAG with sufficient resources, the earliest finish of a node is its duration plus the latest predecessor finish. The latest required sink gives the schedule's completion time. Real resources can serialize otherwise independent nodes or introduce contention and queuing. Add the corresponding constraints or describe the DAG result as a lower bound. A trace gives one observed schedule, not all counterfactual schedules after optimization. Removing an operation can change resource pressure and the chosen kernel path, so preserve the model's assumptions when making predictions.

An operation can consume substantial aggregate device time while having slack before the next required join. Shortening it within that slack need not change step completion. Conversely, a small exposed communication tail at the end of backward execution can lie directly on the finish path. Inspect launch readiness and the consuming operation, not only the total size of a category. A ranked operator table helps select candidates, but its inclusive entries may overlap each other and cannot safely be added as independent pieces of wall time.

Even in a fixed ideal DAG, speeding up the currently longest branch can make a previously shorter branch become critical. A projected saving therefore saturates when another required path catches up. For a single shortened node, the graph's finish reduction cannot exceed the amount removed from that node, and it may be zero. If an experiment saves more, investigate changed scheduling, contention, work, or measurement boundaries rather than claiming the fixed-duration DAG predicted that effect. Counterfactual reasoning is useful precisely because its assumptions are visible.

Capture enough adjacent steps to distinguish steady execution from initialization, compilation, allocator growth, and data-loader startup. Correlate operator names with stream dependencies and required collective completions. State whether gradient accumulation spans several microbatches before the optimizer boundary. After a change, verify equivalent valid work and numerical behavior, then compare the observed finish with the predicted saving. A shorter kernel that moves a wait elsewhere is not yet an end-to-end improvement. Preserve the before-and-after trace and the hypothesis it was meant to test.

The roofline lab supplies a separate resource ceiling rather than a scheduler. Reset, set both intensity I=2 FLOP/byte and memory bandwidth B=100 decimal GB/s, predict 200 GFLOP/s, then Play. Reset and set both I=4 and B=100 to predict 400. Its fixed compute roof is 1000 GFLOP/s and playback traces that roof without changing the chosen kernel. The instrument contains no training DAG, launch queue, collective, or completion boundary. Use its ceiling to ask about a resource constraint; use the dependency calculation to explain when improving that resource can shorten the step.