Fill-drain schedules and bubbles

Label a forward operation by stage and microbatch. The first stage can process another microbatch while the next stage works on the previous one. A downstream forward cannot start until its own input arrives, so later stages begin with idle intervals. In the all-forward schedule, the backward phase begins after the forward wave finishes, then travels in the reverse direction. A stage’s backward also requires its matching saved or recomputed activation state, not just any received gradient.

For the following arithmetic, assume P stages, M microbatches, the same forward duration f on every stage, and the same backward duration b on every stage. A stage cannot run two modeled compute operations simultaneously; communication, update time, recomputation, and other overhead are omitted or already handled consistently. Forward and backward durations need not equal each other. These restrictions are what make the simple closed form useful. A partition with unequal costs or substantial exposed transfers needs an explicit dependency timeline instead.

The additional time over ideal computation is the bubble time. Dividing it by ideal time gives an overhead ratio, while dividing it by elapsed iteration time gives an elapsed idle fraction. Those numbers differ, especially when M is small. Under the balanced compute-only assumptions, average device compute utilization equals the useful fraction of elapsed time. It is not a profiler utilization reading and does not include simultaneous communication. Label the denominator rather than reporting an ambiguous bubble percentage.

Increasing the microbatch count reduces the modeled relative bubble, but it is not free. At a fixed global batch, smaller microbatches can change kernel efficiency and communication startup costs; at a fixed microbatch size, more microbatches increase the global batch and can change optimization behavior. An all-forward schedule can also retain activation state for all M microbatches. Choose the experiment’s invariant first, then measure the consequences of the change rather than attributing every throughput difference to the closed-form bubble term.

Accumulate the correctly normalized microbatch gradients while the parameter version remains fixed, then perform the intended optimizer update after all required backward and gradient-reduction work completes. This ordering supports synchronous training semantics under additional loss, randomness, state, and arithmetic assumptions. It does not automatically make microbatch batch-normalization identical to full-batch statistics or floating-point summation bitwise identical. Distinguish optimizer equivalence from a favorable utilization curve, and include numerical tests alongside schedule timing.

The queue lab illustrates only cessation of admissions. Reset, set arrival rate to 4 and service rate to 4, predict backlog at one second, then Step twice: it remains four. For a separate drain experiment, Reset and explicitly set both arrival rate to 0 and service rate to 4 before two Steps: it reaches zero. The model has no stages or forward/backward waves, so its one-second drain cannot be substituted into a training bubble equation.