Pipeline weight-version consistency

A backward derivative must use the parameter values and forward state required by the forward computation it differentiates. If a stage overwrites its weights before that backward, reading the latest values can produce an invalid mixture. Track a version identifier when the stage executes each forward, retain or reconstruct the required version, and use it for the matching backward. Saved activations alone do not protect against a changed parameter multiplying an upstream gradient. The required values depend on the derivative, not on whether tensor shapes still match.

Original PipeDream stashes weight versions so that a microbatch’s forward and backward at a stage use the corresponding local version. It can keep the pipeline moving while updates occur, but microbatches can encounter versions from different stages and updates can apply delayed gradients. A local consistency guarantee therefore differs from asserting that the entire gradient was evaluated at one current global parameter vector. The paper’s optional vertical synchronization aligns versions across stages, but still applies delayed gradients and does not become GPipe-style accumulation. Version stashing consumes memory and needs retirement rules after every dependent backward and any checkpointed recomputation finish.

In a synchronous flush, all microbatch contributions for the intended update use the fixed parameter vector, are normalized correctly, and complete any required cross-replica reductions before the optimizer advances. That can match the chosen synchronous reference subject to random-state, batch-dependent-operation, and arithmetic assumptions. An asynchronous update can instead apply a gradient evaluated using older or stage-specific versions to newer parameters. Such an algorithm can be useful, but its convergence and quality must be evaluated as that algorithm rather than inferred from a throughput improvement.

The distinction also matters when recovering a checkpoint. A synchronous checkpoint at a drained update boundary can record one consistent optimizer state and data position. An in-flight asynchronous checkpoint may need weight-version tables, pending microbatch states, and update order to resume its exact semantics. Dropping those details can silently change the algorithm even if every parameter tensor reloads successfully. Record the schedule, version policy, and optimizer boundary in experiment metadata so an apparent numerical regression can be traced to its actual execution contract.

Choose a two-stage scalar model with an analytically known gradient, then deliberately change a parameter between forward and backward. Compare three cases: a valid fixed-version backward, an invalid mixed-version derivative, and a valid but stale gradient applied to newer state. These have different meanings even when a training loss happens to decrease in all three. Extend the test to checkpoint replay, which must use its matching parameter version as well as its matching random-state behavior.

The queue lab offers only an analogy for outstanding older work. Reset, set arrival rate to 3 and service rate to 3, predict backlog at one second, then Step twice: four units remain. Compare a fresh Reset with both arrival rate at 0 and service rate at 4 before two Steps: zero remain. The instrument contains no weights, version identifiers, derivatives, or optimizer updates. A drained fluid queue cannot certify that a real checkpoint or gradient is version-consistent.