Replicated training: agree on the gradient and the update

DDP does not choose which examples each worker receives. The input pipeline or sampler must establish the intended global batch, including duplicates, padding, and dropped examples. Each worker computes a local loss and its gradient at the current replica parameters. Under the stated default reducer, corresponding gradients are summed across the group and divided by the group size. Equal-size local mean losses therefore produce the global mean gradient for a separable per-example objective. Local sums instead produce an average of sums, which has a different scale from one global sum.

Identify the objective before adjusting the learning rate. A missing division and a deliberately changed global batch are different changes, even if either can be disguised by one learning-rate setting in a simple SGD experiment. Regularization terms, gradient clipping, momentum, and adaptive optimizer state make that disguise unreliable. Use small deterministic gradients to expose the reducer’s scale directly. Changing world size without revisiting local loss reduction can change the mathematical update while throughput and reported local losses still look reasonable.

Default initialization synchronization establishes matching model parameters when enabled; disabling that synchronization transfers responsibility to the application. During ordinary training, DDP does not repair arbitrary parameter divergence with a parameter broadcast after every optimizer step. Replicas remain aligned because they apply equivalent updates to matching state using the synchronized gradients. An optimizer restored differently on one worker, a rank-dependent clipping decision, or a scheduler advanced on only one worker can break this reasoning even when the reducer returned matching gradients.

Non-parameter buffers have a separate contract, including configured buffer broadcasts. Batch normalization statistics and stochastic model behavior also matter when comparing a distributed run with a single global-batch reference. A rank-local batch-dependent operation need not equal the same operation on concatenated inputs. Start an equivalence fixture with a separable deterministic model, then deliberately add such features with their intended semantics stated. Agreement among replicas and equivalence to a particular reference objective are related but distinct properties.

Every required collective must have matching participants, compatible tensors, and compatible order. A faster worker cannot safely decide to skip synchronization merely because its local loss is small. Likewise, a communication future being created does not mean the reduced buffer is already safe for an optimizer to consume or overwrite. Use the framework’s documented completion and stream dependencies. Log rank, step, tensor metadata, and the first mismatch when debugging, rather than infer correctness from workers reaching similar wall-clock timestamps.

The optimization lab isolates an update-scale consequence. Reset, explicitly set step size 0.1 and initial position two, predict the first iterate, then Step once. Compare a fresh Reset with step size 0.2 and initial position two explicitly set, again predicting before one Step. For its fixed objective x²/2, the iterates are 1.8 and 1.6. Doubling an effective gradient scale resembles doubling this toy step size, but the instrument has no replicas, all-reduce, optimizer state, data pipeline, or distributed completion semantics.