Define the next-token task before counting a training example
Specify shifted targets, valid-loss positions, document boundaries, and the conditioning context of a reproducible autoregressive objective.
On this page
The question for today
What you’ll understand
- Identify the target predicted by each input position.
- Distinguish document-reset and cross-document continuation objectives.
- Compute masked mean negative log-likelihood with an explicit denominator.
Autoregressive pretraining predicts a token from an allowed prefix, but that short description leaves important implementation choices unresolved. The tokenizer defines the symbols, a target shift defines which symbol each logit predicts, and document policy determines which earlier tokens may influence the prediction. Padding and excluded transitions determine the loss denominator. Before scaling a run, write a tiny example in which every prediction can be inspected. A smooth loss curve is not evidence that these choices match the intended objective.
Write the shift and the eligible target set
For a document serialized as [BOS,a,b,EOS], the logit at BOS predicts a, the logit at a predicts b, and the logit at b predicts EOS. In this example BOS is conditioning context, EOS is a scored target, and the EOS input position has no further scored target. Those are declared choices rather than universal properties of special tokens. A generic cross-entropy function consumes logits and supplied labels; it does not itself decide the language-model shift or the document policy.
Use a binary eligibility indicator for every candidate prediction, then accumulate both its negative log-likelihood and its count. Ignored padding labels should not dilute the reported mean. The examples here use unsmoothed, unweighted hard-token targets and natural logarithms; label smoothing or class weighting would change the objective or denominator semantics. In PyTorch 2.9, ignore_index applies to class-index targets. A batch with zero eligible targets needs an explicit skip or error policy instead of a division that produces a misleading number.
Choose what happens at a document boundary
One valid policy treats documents as independent sequences, using a fresh BOS and preventing attention to other documents. Another deliberately trains on a concatenated stream with boundary markers and permits earlier documents as context. These are different conditioning distributions. Inserting EOS alone does not enforce a computational reset: if the attention mask still permits earlier keys, information can cross that boundary. Resetting position numbers alone also does not block those keys. The manifest must identify the actual policy used by the model.
Decide whether a transition between documents is scored. Under the independent policy above, EOS from document A must not be paired with document B's BOS as an additional target. The first content token of B is instead predicted from B's BOS under its own context. For long documents split across blocks, separately state whether context is carried, truncated, or overlapped and which targets are scored once. These policies change both exposure and validation loss, so silently switching them mid-run can resemble a model-quality change.
Test leakage by changing forbidden context
Construct a two-document fixture with distinctive tokens and turn off stochastic training effects for the comparison. Under independent masking, changing A should leave the evaluated predictions in B unchanged within the chosen numerical oracle. Changing an allowed token inside B should be capable of changing later B predictions. Inspect the actual shifted labels and masks as well as outputs; a test that accidentally scores no B tokens can otherwise pass a supposed isolation check without exercising the intended property.
The network lab provides a two-outcome probability for inspecting one target contribution. Reset, set both input x=1 and weight w=1, predict sigmoid probability about 0.6225 and positive-target loss about 0.4741 nats, then Play or Step. Reset and set both x=1,w=2 for probability about 0.8176 and loss about 0.2014. The instrument has one fixed-bias sigmoid, not a vocabulary softmax, document mask, tokenizer, or pretraining loop; it illustrates only how an assigned target probability enters a log loss.
Work it through
Average only the three specified predictions
Three eligible shifted targets receive probabilities 0.5, 0.25, and 0.8. Two other storage positions are padding or excluded boundary transitions. Use natural logs, no smoothing or weights, and positive probabilities.
Compute the three target contributions: −ln(0.5)≈0.693147, −ln(0.25)≈1.386294, and −ln(0.8)≈0.223144 nats. The excluded positions contribute neither loss nor count.
The sum is −ln(0.5×0.25×0.8)=−ln(0.1)=ln(10)≈2.302585 nats. Divide by three eligible targets to obtain approximately 0.767528 nats per token.
Dividing by five storage positions instead would report about 0.460517, artificially lowering the mean without changing any prediction. Record the valid count with the numerator so the reduction can be audited.
The declared objective reports about 0.767528 nats per valid token. A different boundary or special-token policy would require rebuilding the eligible target set.
The living notebook
Explore the model
Interactive lab · network
Turn one target probability into one loss contribution
Fresh Reset → set both x=1,w=1 predicts p≈0.6225 and −ln p≈0.4741; Reset → set both x=1,w=2 predicts p≈0.8176 and −ln p≈0.2014. Play or Step shows the scalar primal. No tokenizer, vocabulary softmax, document mask, or training update is executed.
- Product · wx
- 1.2
- Logit · wx − 0.5
- 0.7
- Sigmoid output
- 0.6682
One inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA token loss needs both the target probability and a precisely counted eligible position.
Pause & reason
Check your understanding
Your turn
Try it yourself
Two independent documents are [BOS,a,b,EOS] and [BOS,c,EOS]. BOS is context only, EOS is scored, and EOS-to-next-document transitions are excluded. How many valid next-token targets exist, and which target begins the second document?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
List each input-to-target pair within a document rather than subtracting one from the length of the entire packed storage block.
Compare with the solution
The first document scores a, b, and EOS: three targets. The second scores c and EOS: two targets. There are five valid targets in total, and c is predicted from the second document's BOS under its own allowed context. Counting the packed seven tokens minus one would incorrectly add an inter-document transition. Padding to a larger storage block would not increase the valid target count.
Keep the idea
Take it with you
- Shifted labels and eligible positions define the implemented prediction task.
- EOS and position resets do not themselves enforce attention isolation.
- Report summed negative log-likelihood together with the valid target count.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.