Interleaved pipeline schedules

With four physical ranks and two chunks per rank, a simple looped placement assigns virtual stages zero through three to ranks zero through three, then stages four through seven to those same ranks again. One microbatch must follow virtual order, including the boundary from virtual stage three on rank three to virtual stage four on rank zero. Two virtual stages on the same rank cannot both claim the full device compute resource at the same modeled time. The schedule chooses among their ready operations.

Each virtual chunk has its own activation identities and backward dependencies. A message needs to identify the logical boundary, microbatch, and direction rather than just a physical neighbor. A rank can receive traffic for different chunks, so a buffer indexed only by peer rank can be reused incorrectly. The local priority rule matters as well: depth-first and breadth-first policies choose different ready work. PyTorch 2.8 documents interleaved 1F1B’s preference for earlier microbatches when several local stages are ready.

The original balanced analysis splits each device’s previous forward and backward work into v equal chunks. Under its legal interleaved order and ideal communication assumptions, the bubble duration is divided by v while useful per-device computation remains unchanged. That result is about a specific balanced model; smaller chunks can be less efficient, and their extra boundaries can expose communication startup and bandwidth costs. A nominal reduction in idle compute does not establish a shorter measured iteration when the link becomes the bottleneck.

The original Megatron schedule requires a microbatch count divisible by the physical pipeline size. PyTorch 2.8 describes a flex scheduling rule that relaxes that original restriction, with its own admissibility condition. Do not impose a paper’s constraint on every later runtime or assume every arbitrary count is legal. Validate the chosen schedule constructor and version, then trace its warmup and cooldown. A zero-bubble label on a different schedule also has separate backward-partition and timing assumptions and is not a property inherited by all interleaving.

Count boundary tensors explicitly for the selected layout. In a simple chain with equal-size boundary payloads, more virtual stages mean more cross-rank sends for each microbatch, including wraparound transfers. Actual volume depends on tensor shapes and placement rather than merely multiplying a physical-device count. Include communication-buffer lifetimes and possible contention with tensor-parallel collectives. A useful comparison holds the model, global batch, precision, and optimizer semantics fixed while reporting changes in chunks, message count, memory, and end-to-end time.

The modular lab visualizes only the cyclic rank arithmetic. Reset, set modulus to 4 and additive step to 1, predict the period, then Step four times to return to zero. Compare a fresh Reset with both modulus at 4 and step at 2 before two Steps; only two residues are visited. This arithmetic explains why a placement stride can omit ranks, but it has no virtual-chunk operations, data dependencies, backward work, messages, or device resource conflicts.