Scaling fits and compute allocation

A common illustrative loss fit combines an irreducible floor with terms that decrease as model size and training data increase. Parameters are learned from measured runs; the formula does not establish that data are independent, equally useful, or indefinitely available. The meaning of loss also depends on tokenizer, validation distribution, and preprocessing. Compare held-out fits on a consistent task, and retain residual plots so systematic misses are visible rather than concealed by a smooth line.

Training compute is often approximated as proportional to parameter count times processed tokens for a chosen dense model family. The proportionality constant hides architecture and accounting conventions. Attention cost, sequence length, sparsity, recomputation, and hardware utilization can break a naive conversion from floating-point operations to wall time. Distinguish algorithmic operation count from measured accelerator-hours and cost. A budget in money may favor a different model than a budget in abstract operations.

Substitute the compute constraint into the loss model, leaving one variable. At an interior optimum, increasing model size improves the parameter-limited term while worsening the data-limited term because fewer tokens fit. Balancing derivatives gives a conditional allocation rule. If the fitted exponents coincide and units are normalized consistently, the optimum divides the tradeoff symmetrically only when the fitted coefficients also align. Equal exponents alone do not imply equal raw parameter and token counts.

Use the optimum as a candidate for experiments, not a purchase order. Propagate uncertainty in fitted coefficients and compare several nearby allocations. A shallow loss valley may make a range of choices indistinguishable given run-to-run noise. Validate the extrapolation with intermediate scales before spending the full budget. Data repetition, deduplication, or a changed mixture can alter effective data quality even when the nominal token count increases.

A training-optimal model need not minimize lifetime cost. A smaller model trained longer can cost more to create but less to serve across billions of future requests. Count adaptation, evaluation, storage, and inference under the same workload and quality requirements. Conversely, a larger model may reduce the number of calls or generated tokens needed for a task. Compare completed useful tasks, not just a token price detached from output quality.

Capability benchmarks can show thresholds, noise, or metric artifacts even when validation loss changes smoothly. An empirical loss fit does not prove that a deployment will become safe, reliable, or capable at a particular scale. Record the forecast, uncertainty interval, and decision criteria before running the next scale. When evidence disagrees with the prediction, inspect data and optimization changes instead of forcing a new result into the old curve.