Freeze the tokenizer as part of the experiment

Train the tokenizer on a documented sample drawn from the permitted training partition, preserving the sample identities or reproducible selection rule. Do not fit vocabulary or normalization statistics on the final held-out evaluation text and then describe that evaluation as untouched by representation selection. Tokenizer fitting is not the same as updating language-model weights, but it still adapts the experiment to data. Development data may support declared design choices; reserve a separate final test when independent assessment is required.

SentencePiece's original paper and pinned v0.2.0 documentation describe trainable subword models and normalization as parts of a reusable tokenizer artifact. The BPE paper explains learning subword units from corpus statistics. Their existence does not prescribe a universally optimal vocabulary size or domain sample. Inspect coverage across scripts, code, whitespace, and malformed-input policy using permitted diagnostic material. Keep added special-token IDs and encode/decode behavior in fixtures, since changing those after training can reinterpret model embedding rows.

A larger vocabulary allocates more rows to input embeddings and often expands the output projection. It may shorten some serialized documents by representing frequent strings with fewer tokens, but the effect varies by language and domain. A smaller vocabulary can reduce those rows while increasing sequence length or fragmenting rare scripts. Measure the actual encoded lengths and unknown or fallback behavior on allowed diagnostic sets. The raw number of characters per token is an average, not an admission guarantee for every document.

Within a dense attention score calculation with fixed width and one sequence, reducing length from four thousand to three thousand changes its quadratic position-pair count. That calculation does not imply the entire training step becomes equally cheaper: feed-forward work, softmax over vocabulary, kernel choices, block packing, and memory overhead can change differently. Fixed token-budget comparisons also expose different amounts of raw text when tokenizers differ. State which resource is held fixed before crediting an improvement to representation quality.

Save the tokenizer artifact, its training configuration, library version, corpus sample reference, normalization policy, vocabulary and special-token mapping, and a digest for integrity comparison. Include representative round-trip fixtures under the documented normalization contract; lossless recovery of every original byte should not be promised if normalization intentionally changes text. Changing a tokenizer after a model has trained usually requires a deliberate compatibility or retraining plan rather than merely replacing a filename. A familiar-looking token string can still map to a different row.

The cache lab isolates lookup reuse under a fixed, artificial identifier stream. Reset, set both capacity C=4 and stride s=4, predict twelve hits over the sixteen initial requests, then Play or Step. Reset and set both C=4,s=1 for a zero-hit comparison with more distinct identifiers. Playback caps at 4,096 requests. This can motivate inspecting embedding-access patterns, but the model has no learned vocabulary, token frequencies, merge rules, variable-sized embedding rows, or processor-cache hierarchy; it cannot choose a tokenizer for you.