Autoregressive decoding and sampling

Autoregressive generation conditions each new token on the input and all previously selected output tokens. Greedy decoding takes the highest-probability token at each step, which is a local decision. It does not generally maximize the probability of the complete sequence because an initially less probable token can lead to a much more probable continuation. Beam search retains several candidate prefixes, but its sequence scoring and stopping policy still encode choices about length and preferred outputs.

For an open-ended writing task, maximizing likelihood can produce repetitive or uninteresting continuations. For structured extraction, sampling diversity may be unnecessary. Choose a decoding policy for the task and compare complete outputs under a fixed evaluation protocol. The log probability of a sequence is easy to add across tokens, but comparisons across different tokenizations or lengths need care. A model can assign high probability to a common misconception; decoding alone cannot make its learned distribution factually reliable.

For positive temperature, divide logits before softmax. Temperatures below one sharpen relative preferences; above one flatten them. Temperature approaching zero concentrates mass on maximal logits, with ties requiring a convention. A serving interface may implement a special greedy mode at zero, but division by zero is not the mathematical definition. Raising temperature is not an independent source of knowledge: it changes which available continuations are sampled and can increase error as well as variety.

Top-k retains the k highest-scoring tokens and renormalizes. Nucleus, or top-p, sampling retains a smallest high-probability prefix whose cumulative mass reaches the chosen threshold, then renormalizes. Unlike a fixed k, its retained set can vary with distribution shape. State the order if combining temperature and truncation because it can change the selected set and resulting probabilities. Tie handling and minimum-token rules are implementation details that belong in a reproducible decoding configuration.

An end token, stop string, length cap, or cancelled request can each terminate a continuation. Hitting a cap is truncation rather than successful completion and should be recorded separately. Stop strings can cross token boundaries, so process decoded text carefully. Also distinguish time to first output from time to a complete useful answer. A policy that produces much longer answers can raise total cost despite unchanged per-token throughput, and an empty answer can look fast without completing the task.

Record model and tokenizer revisions, prompts, sampling parameters, seeds, output limits, and relevant runtime versions. A seed makes a random process repeatable under controlled conditions but does not promise identical distributed GPU results across every kernel or scheduler. Compare stochastic policies with repeated samples and task-level uncertainty. Inspect failure classes as well as aggregate scores, including repetition, premature termination, invalid formats, and unsupported assertions. Good decoding settings are evidence-backed choices for a workload, not universal constants.