Define the schedule clock and every endpoint

Here u numbers successfully applied optimizer updates from one through U, and η(u) is the rate used by update u. Accumulation microbatches do not each advance this clock. Under this explicitly chosen policy, an overflow-skipped update does not advance u, though consumed tokens and attempted updates are still counted separately. Other experiments can schedule by attempted updates or valid tokens, but that is a different policy and must state what happens when a step is skipped or replayed.

Token-based scheduling can make exposure boundaries more comparable when valid-token batches vary, but it also needs an exact counter definition. Consumed tokens that failed to influence an applied update are not automatically the same as accepted trajectory exposure. Record schedule configuration, current progress, and the last applied rate in checkpoints. If the batch policy changes, a fixed number of updates no longer implies the same number of valid targets, so comparisons should report both clocks rather than silently renaming one as the other.

Use a linear warmup for updates one through W, reaching ηmax at update W. For subsequent updates through U, use one half-cosine down to ηmin at U. Require integers 1≤W<U and nonnegative rates ηmin≤ηmax. No update zero is applied in this example, and behavior after U is deliberately outside the scheduled run. These choices remove ambiguities that a bare pair of hyperparameters cannot resolve, including whether the first update has zero rate.

The original Transformer paper used an inverse-square-root schedule with warmup, showing that cosine is not a universal architectural requirement. The pinned Transformers 4.57.1 implementation provides concrete schedule variants with explicit counter formulas, while PyTorch 2.9 documents optimizer/scheduler call ordering. Our formula is an independent stated example, not a promise that every named cosine helper uses the same endpoints. Inspect the selected helper's actual rates, including initialization, instead of equating a function name with this convention.

Log rates used by the first few updates, the last warmup update, the first decay update, and the final update. Test an overflow skip with a controlled fixture and confirm the chosen clock policy, then restore a checkpoint and compare the next applied rate. Also inspect every optimizer parameter group when rates differ. Reconstructing a scheduler from its defaults while restoring only weights can restart warmup or advance decay incorrectly, even when the resumed loss initially looks plausible.

The optimization lab isolates constant-step sensitivity on a quadratic, not this schedule. Reset, set both step η=0.5 and nonzero initial position x0=2, predict contraction, then take four Steps to reach x4=0.125. Reset and set both η=1.5,x0=2 for alternating contraction to the same fourth value. Each Step is one recurrence update. The lab has no warmup, cosine decay, Adam state, skipped updates, or token clock; it explains why the actual applied step sequence deserves inspection without validating a pretraining schedule.