Strong and weak scaling answer different fixed-work questions

For strong scaling, preserve a defined global work unit while increasing resources. In training this can mean the same global effective batch, sequence distribution, model, and valid optimizer update. Local microbatches and accumulation may change to distribute that work, but record those changes because they can alter kernel efficiency and communication timing. Smaller local matrices may use devices less efficiently even when the mathematical work is identical. A global batch reduced along with device time is not a strong-scaling improvement for the original completed work unit.

Choose a feasible baseline with device count D0 and measure completed time for the same W on D devices. The ideal resource-proportional speedup is D/D0. Efficiency divides the measured speedup by that factor. It can occasionally exceed one when memory, cache behavior, or algorithms change favorably, but such an observation needs explanation rather than a universal linear-scaling extrapolation. Preserve hardware topology and placement information; doubling devices within one fast interconnect domain is not necessarily comparable with doubling across another network boundary.

A weak-scaling experiment increases global work in proportion to device count so a declared per-device work amount remains similar. Ideal completed time stays constant. Specify the work model: holding tokens per device may not preserve arithmetic when sequence lengths, attention shape, routing, or model architecture change. Communication can still grow with participant count or topology. Report the exact global work and observed time instead of calling every fixed-local-batch experiment equivalent regardless of its shapes and synchronization structure.

Increasing global batch changes an optimization trajectory even when one step remains fast. Learning-rate schedules, gradient noise, number of parameter updates per token, and time to a quality threshold can all differ. Weak-scaling efficiency is therefore a system measurement for a defined workload family, not a guarantee that training reaches the same quality sooner. Compare quality and total valid work to a target separately. For model-size scaling, define another explicit family rather than silently combining larger models with a token-based weak-scaling claim.

Measure communication, input readiness, pipeline bubbles, kernel shapes, and rank tails at each scale. A fixed host overhead becomes a larger fraction as useful compute per rank shrinks. Extra ranks can introduce longer collectives or increase the chance of a required slow arrival. Use the dependency path to predict the next limit, then test it rather than fitting one favorable scaling line through two configurations. Include total device-time cost and memory feasibility with wall time; a faster step can consume more resources per valid work unit.

The roofline lab illustrates a bounded resource response rather than device-count scaling. Reset, set both I=8 FLOP/byte and B=50 decimal GB/s, predict 400 GFLOP/s, then Play. Reset and set both I=8 and B=100 for 800. Reset and set both I=8 and B=200 for the fixed 1000 compute cap rather than 1600. The instrument changes a hypothetical memory-bandwidth parameter, not the number of GPUs or work per rank. Use the saturation as a resource-ceiling example; obtain strong and weak scaling from real fixed-work and growing-work experiments.