Tokenization and vocabulary design

Word vocabularies make frequent words compact but struggle with unseen names and productive morphology. Character or byte representations cover a smaller alphabet while producing longer sequences. Subword methods occupy the middle: common fragments can have their own identifiers, while less common text decomposes into smaller pieces. Byte-pair encoding repeatedly merges frequent adjacent units during vocabulary construction. Applying a learned tokenizer uses its fixed vocabulary and rules; it does not relearn merges from every incoming request.

SentencePiece describes tokenization directly from raw text without requiring a language-specific word splitter. Its implementation supports different segmentation models; the library name is not synonymous with one algorithm. Distinguish vocabulary coverage from preservation of the original input. A normalizer can merge visually similar or canonically related strings, and a tokenizer without an appropriate fallback may emit an unknown symbol. Test round trips under the declared normalization policy, including whitespace, combining marks, code, and unusual scripts.

A larger vocabulary can shorten common sequences but enlarges the input embedding and often the output projection. With tied input and output weights, one matrix can serve both roles; with untied weights, account for both. Vocabulary size also affects the logits produced at each decoding step. Shorter sequences reduce cache growth, but they do not establish an unconditional speedup because kernels, batching, and model dimensions influence the actual bottleneck. Measure complete requests rather than advertising one compression ratio.

Compare token counts on representative languages and domains, not only English prose. The same semantic task may require very different token budgets across scripts, source code, or technical notation. A context limit counts all serialized input, including instructions, tool definitions, retrieved passages, and conversation delimiters. Reserve output space explicitly. A character-per-token average is useful for rough planning, but admission control must encode the actual request with the exact tokenizer to avoid truncation or rejection.

Store tokenizer files, normalization settings, special-token identifiers, and conversation-template revision alongside weights. A token identifier has meaning only relative to that vocabulary. Adding a new token also requires a compatible embedding row and a training or initialization policy; simply changing the text mapping cannot teach a pretrained model the new symbol's meaning. During deployment, compare golden input strings and exact token identifiers between training, offline evaluation, and serving implementations.

Treat truncation as a product decision. Removing the oldest conversation turn can erase a constraint; cutting a document can remove the evidence needed to answer. Apply a deliberate retention policy, count tokens after final formatting, and report when evidence is omitted. Benchmark segmentation length by language and task while also measuring output quality. A tokenizer that saves memory but systematically loses required distinctions is a representation regression, even if aggregate throughput improves.