Training cost: define useful token contributions

Specify what the objective actually weights. An input position can be necessary context without being a supervised target, and a zero-loss mask can still require attention computation. For our accounting example, useful tokens are target contributions accepted into the intended optimizer history and retained across recovery. Intentional later epochs can contribute again because the planned objective revisits data; useful token count is not the number of unique corpus strings. Document the loss, masking, weighting, and progress definition before publishing a token-efficiency ratio.

Classify excluded positions with disjoint categories when subtracting them from a processed count. A padding position already included in the padding category must not also be subtracted as masked. Failed or replayed contributions need lineage so work is charged every time it executes but credited once to accepted progress. A checkpoint boundary is one defensible accounting convention, not proof that every update before a crash is invalid for all purposes. State which state is recoverable and how the measured numerator and denominator close over the same run segment.

Resource-time includes initialization, data stalls, evaluation, checkpoint writes, recovery, and idle allocations within the selected boundary. Add separately priced storage or transfer operations only if they are not already included in another total. Failed intervals can increase this numerator while adding no accepted useful tokens. A throughput optimization that merely moves evaluation or checkpointing outside the reported window can look cheaper without changing the full run. Preserve the complete experiment lifecycle and state which shared infrastructure allocations are included.

Global token normalization also matters to the comparison. Unequal worker token counts do not justify averaging worker means as though each represented equal objective weight. A cheaper recipe with changed gradient weighting is a different training experiment unless that change is intended and evaluated. Hold quality targets, optimizer behavior, and recovery semantics appropriately constant when comparing implementations. Report both raw throughput and useful contribution throughput, alongside final evaluation; none of those alone establishes that two training runs deliver equivalent models.

Track padding by batch shape, loss masks by objective, and discarded work by failed attempt and recovery lineage. Validate aggregate counts against a small trace where every position and optimizer update can be inspected. Average valid fraction can hide a workload class whose long sequences pad poorly, or a failure pattern that repeatedly loses expensive late-stage work. When estimating from samples, define the sampling unit and dependencies; adjacent tokens in one packed sequence or one failed worker are usually not independent observations.

The sampling lab illustrates an intentionally independent valid-position model. Reset, explicitly set p=0.8 and n=20, predict, then Play or Step. Repeat from a fresh Reset with p=0.8 and n=80 explicitly set. For independent Bernoulli positions, the displayed sample-proportion variances are 0.008 and 0.002. Its mean p is a known input, and playback highlights a fixed exact distribution rather than collecting a real trace. It models no optimizer, masking logic, checkpoint loss, token correlation, cost, or model quality; use a sampling design suitable for the actual run.