Compare validation loss on a fixed, counted evaluation contract

Record evaluation document identities, tokenizer artifact, serialization, truncation or sliding-window policy, target eligibility, document boundaries, and numerical evaluation mode. The same raw text can have different negative log-likelihood when context resets more often. If overlapping windows are used, distinguish context-only tokens from newly scored targets so repeated context is not counted as new evaluation evidence. Disable or explicitly control stochastic effects and state whether the result is deterministic within the chosen environment.

Keep evaluation material separate from tokenizer fitting and model training according to the declared split. Development losses can guide hyperparameters, but repeated selection adapts the experiment to that development set; a final untouched test serves a different role. An evaluation contamination scan can identify known overlaps under its method, not prove the absence of every semantic duplicate. Attach the actual scan scope and corpus versions rather than turning a clean report into an unrestricted no-leakage guarantee.

Perplexity for a natural-log token loss is exp(L). The pooled perplexity is therefore a weighted geometric combination of domain perplexities, because pooling happens in log-loss space. An arithmetic average of perplexities generally gives a different number. Retain sufficient precision in summed losses and counts before exponentiation, and do not mix bits with nats. If losses are measured in bits, the corresponding base is two rather than e.

Cross-tokenizer perplexities are not directly comparable as though tokens were identical units. Even with one tokenizer, changed context-window stride or boundary rules can alter the task. A raw-byte-normalized description can sometimes support another declared comparison, but it still needs a precisely defined serialization and likelihood calculation; it is not obtained by casually relabeling per-token perplexity. For quality assessment, report domain behavior and relevant controlled evaluations alongside the aggregate rather than claiming that one lower number settles every use case.

Report the observed token-weighted validation mixture and, when useful, a separately fixed reporting mixture. These answer different questions. If domain losses remain constant while the report shifts toward a harder domain, the pooled loss can rise with no within-domain regression. Conversely, a large easy domain can hide a decline on a small important domain. Compare the domain values first, then state the weighting decision that produces the headline number.

The probability lab provides a simple two-domain count model under hypothetical independent assignment. Reset, set both p=0.1 and n=20, predict two expected assignments to the minority domain, then Play or Step. Reset and set both p=0.5,n=20 for ten expected assignments. The exact binomial bars describe count uncertainty under that model, not token losses, perplexity, document correlation, or a real held-out sample. Actual validation weights should come from measured valid-token counts, not from this toy expectation.