Sharded checkpoint resharding

Give every logical tensor a stable name and semantic shape. Record how stored fragments map to its global indices, including dimension order, datatype, padding rules, and any split or merge transform. Parameter aliases and tied weights need an explicit policy so multiple names do not become divergent independent values. Optimizer moments must use the same logical parameter identity even when internal flattening or parameter object IDs change. A successfully opened file says nothing about whether its contents belong to the intended layer or moment.

The destination creates its supported model and optimizer layout, then asks for its required logical regions. PyTorch 2.14 DCP operates in place and uses destination state together with checkpoint metadata for supported load-time resharding. Megatron’s sharded mappings include global shape and offsets. Those capabilities are format- and state-specific: arbitrary Python objects, unsupported optimizer encodings, or architecture changes do not become transformable just because tensor resharding works. Check compatibility for every state category and software version before committing to a new allocation.

For a contiguous one-dimensional illustration, an overlap length is the positive part of the smaller end minus the larger start. Half-open intervals make adjacent regions unambiguous: their shared boundary is not a duplicated element. In multidimensional or strided layouts, a planner must preserve shape and stride meaning rather than assume an overlap is one contiguous byte range. A flattened optimizer format may need a supported conversion to logical tensors before the same intersection reasoning applies. Do not infer that conversion from an incidental filename suffix.

Output buffers may include alignment padding that is not part of the logical tensor. A converter must avoid treating padded values as real parameters or shifting the next tensor’s offset. Compression and quantization introduce their own decoding rules and may alter numerical equivalence; exact lossless reshaping is a different operation from changing precision. Validate source and destination coverage, dtype, names, and values. Compare selected reconstructed slices or small full tensors to a trusted logical reference before allowing the resumed optimizer to consume them.

A resharding test can fill each logical tensor with an index-derived pattern, save it under one partition, and restore it under another. The oracle checks every global index, tied-weight relation, and optimizer-state association. This catches gaps, duplicates, and axis transposes that a scalar loss can miss. Then test an actual next training update under stated numerical tolerances. Correct tensor reconstruction is necessary, but changed reductions, input allocation, or optimizer settings can still change the subsequent trajectory.

The modular lab illustrates complete index coverage with different visit orders. Reset, set both m=12 and a=1, predict period twelve, then Play or Step. Separately Reset, set both m=12 and a=5, predict period twelve with a different order, then Play or Step. Each additive orbit visits all residues exactly once per cycle. The model does not read shards or implement a tensor planner. Its limited lesson is that position labels must retain their meaning when traversal order changes; equal counts alone cannot identify the intended values.