Cache the exact reference probability question

A cache key should bind reference weight and adapter identities, tokenizer and vocabulary, chat template and special-token rules, exact prompt and response token IDs, truncation policy, attention and completion masks, and the score-reduction convention. Include implementation and numerical settings that affect the accepted reproducibility tolerance. A human-readable model name can move, and two adapters over the same base weights need not define the same reference. Preserve dataset and pair identities so chosen and rejected scores cannot be swapped by an ordering or join error. Content identity and schema checks are more meaningful than trusting a filename called reference.

Reference inference should use the declared evaluation behavior, including dropout and any mutable state. If the reference changes during training, a once-computed score is stale for the new objective unless the algorithm explicitly versions and refreshes it. Pinned TRL 0.23.1 documents precomputation and its reference-model configuration; inspect the chosen variant rather than assume every synchronization option is compatible with fixed cached values. A cache's correctness depends on the actual computation contract, not merely on whether the trainer accepts a column with the expected name.

The model predicts a next token from earlier context, so labels and logits must be aligned after the causal shift. Prompt tokens condition the response even when their likelihood terms are excluded from a completion-only sum. Padding and truncated tokens must not accidentally contribute. Appending EOS or duplicating a special token can change both the probability and which sequence is represented. Transformers 4.56.2 chat-template documentation describes how formatting already supplies required control tokens; tokenizing a rendered template with extra special tokens can duplicate them. Keep the rendered and tokenized artifacts available for small diagnostic examples.

Sequence sums and token means are different reductions. With unequal response lengths, a pair can reverse order after division by length, and substituting a mean into a sequence-ratio derivation changes the objective. Record whether normalization is absent, common across a batch, or specific to each response. Compare the policy and reference paths using the same target mask and reduction. A correct score for another tokenization is still the wrong score for this loss. Zero selected tokens should be an explicit invalid or separately handled case, never an unnoticed divide-by-zero or fabricated zero-length average.

Validate a representative set of short, long, truncated, padded, and special-token examples against the live reference path. Check chosen/rejected joins, finite outputs, dtype, and a stated absolute or relative numerical tolerance. Different kernels and reduction orders may create small differences; a tolerance should reflect the supported implementation and the margin sensitivity, not conceal a mask or identity error. Record the reference scoring artifact and validation results with the training run. Invalidate or regenerate when any influencing input changes, and fail explicitly when a required score is missing or incompatible.

The cache lab isolates retention and reuse under a synthetic address stream. For each separate view, Reset, set both capacity C and stride s=4, predict the initial counts, then inspect. Capacity four gives twelve hits and four misses; capacity two gives zero hits and sixteen misses. This exact LRU instrument contains no model versions, token masks, scores, or validation tolerances. It shows a capacity effect only. The actual preference-training cache must establish semantic validity separately, because a stale score can be returned quickly and repeatedly without the cache noticing its changed meaning.