Topology-changing training resume

Data parallelism usually partitions different examples among replicas, while tensor and pipeline parallelism partition model computation. Expert parallelism introduces additional ownership and routing state. A format that can redistribute one category of shards need not support every combination or arbitrary model changes. Megatron documents distinct optimizer formats with different model-parallel resharding capabilities. Its global tensor mapping cannot make an unsupported legacy optimizer encoding portable. Preserve the original immutable checkpoint while converting so a failed conversion does not destroy the last proven recovery point.

Universal Checkpointing studies a logical representation that separates state from its physical parallel arrangement. That is a capability of a specified framework and supported model mapping, not a promise that any rank-file directory can resume anywhere. Validate architecture, tensor names, shapes, optimizer semantics, and object state. Build the destination configuration first, estimate its memory needs, and use a tested format conversion where required. A full-state consolidation path can have a very different host-memory peak from direct sharded loading, even if final per-device shards fit comfortably.

Under a simple equal-example data-parallel setup, effective examples per update equal replica count times local microbatch size times accumulation steps. Changing replica count while leaving the other factors unchanged changes that batch. Decide whether to preserve it or change the learning policy intentionally. Token-based objectives, variable sequence lengths, masked examples, and unequal rank contributions require actual contribution counts and correct normalization; the simple product is not a universal effective-token formula. Scheduler progress should use the intended committed-step or token convention rather than a newly inferred wall-clock duration.

A new rank assignment can change how samples are partitioned and how rank-local random streams are consumed. Preserve logical sample progress and define how unfinished prefetched work is redistributed. Simply restoring old rank zero’s cursor into new rank zero can repeat or omit data when membership changes. Counter-based randomness keyed by logical sample and operation can help only when the implementation actually defines and preserves that mapping. It does not automatically remove changes from different kernel or collective reduction orders.

First verify reconstructed logical state, then compare the next data identities, optimizer counters, normalized gradients, and parameter update under the new layout. Bitwise matching may fail because reassociated floating-point reductions change rounding, even when the mathematical update is equivalent. Set tolerances from a reference and numerical analysis rather than widening them until every run passes. If changing batch or learning-rate policy intentionally, label the transition and evaluate that new trajectory instead of claiming uninterrupted equivalence.

The optimization lab shows how equal current values can hide different continuation rules. Reset, set both η=0.5 and x₀=2, predict x₂=0.5 and x₃=0.25, then Step three times and inspect both positions. Separately Reset, set both η=1.5 and x₀=2, predict x₂=0.5 and x₃=−0.25, then Step three times. Both runs agree at iteration two but differ at the next update. This fixed quadratic model has no topology, sharding, or distributed optimizer; it supports only the distinction between matching current state and matching future behavior.