Pipeline schedule verification

Identify each operation by stage, microbatch, direction, and parameter version, with separate records for communication initiation and completion. A forward requires its matching input; backward requires its matching local state and downstream gradient; an update requires the intended accumulation and reduction boundary. Buffers move through owned, in-flight, consumable, and reusable states according to actual dependencies. A stage can be locally idle yet globally correct, while an apparently busy stage can run an invalid operation. Treat legality as a prerequisite for performance interpretation.

For a simple chain without recomputation or split backward, count one forward and one backward per stage per microbatch. Count each logical boundary’s forward activation transfer and reverse gradient transfer. Those totals are necessary conservation checks, but equal totals do not prove correct matching or order. Label identities explicitly so a duplicate F0 and missing F1 cannot cancel in the aggregate. Version and lifetime assertions should fail at the first illegal transition rather than waiting for a later loss to diverge.

Compare outputs, loss, all parameter gradients, and one optimizer update against an unsplit reference with the same objective and state. Include unequal token counts, nonmultiple microbatch sizes supported by the chosen interface, and the smallest legal microbatch count. A finite-difference or analytic scalar example offers an independent derivative check rather than another copy of the schedule implementation. Set a justified tolerance for floating-point order, and inspect stateful buffers and random behavior instead of comparing the loss alone.

The version oracle records which weights each forward and backward used, not just the final update counter. A flush schedule should satisfy its common-version boundary; an asynchronous design should satisfy its declared stashing and delayed-gradient rules. A memory oracle verifies that no send or backward reads storage after reuse. A timing oracle checks dependencies and per-device resource exclusion, then compares measured makespan with appropriate lower bounds. A trace simulator can test a stated model, but it cannot establish actual kernel costs or network progress by itself.

Exercise missing participants, delayed transfers, shape mismatches, and deliberately corrupted operation identities in a small harness with bounded observation time. Define whether a failure should raise an error, abort the iteration, or invoke supported recovery. Passing one fault trace does not establish every interleaving safe. Record framework version, schedule configuration, stage mapping, seeds, precision, inputs, and environment so another run can reproduce the same contract. Benchmark only after all required correctness oracles pass, and keep verification overhead distinct from normal training time.

The probability lab illustrates an assumed test-input coverage model only. Reset, set trigger probability to 0.2 and trial count to 5, predict the chance of at least one trigger, then Step. Compare a fresh Reset with both probability at 0.2 and trials at 10 before Step. These are exact binomial probabilities under independent identical triggers, not schedule executions or a theorem about bugs. Correlated tests, unknown trigger rates, and incomplete assertions can invalidate a real coverage inference.