Calibrate on the deployment distribution without leaking evaluation

Use the deployed tokenizer, chat template, preprocessing and model revision while collecting statistics. An activation maximum measured before a template change need not remain representative afterward. Context length matters because longer requests create more opportunities to encounter unusual values, and decoding can differ from prompt processing. A collection of short English questions provides weak evidence for multilingual long-context code generation, even if the token count is large. Record the domain mixture and length distribution instead of reporting only a convenient number of examples.

A maximum observer retains extremes encountered during calibration; a histogram-based approach can trade clipping against resolution under an estimated distribution. Their parameters answer different optimization questions. Rare values can be genuine and important, while a single corrupted example can also stretch a range needlessly. Diagnose the examples creating extremes and check data integrity before choosing a rule. A quantization method that uses reconstruction inputs is still dependent on their coverage, even when no gradient-based model finetuning occurs.

Keep calibration data separate from the final task evaluation set. A development set can select clipping thresholds, group sizes or smoothing settings, but repeated selection turns its results into development evidence rather than an untouched estimate. Group related documents, prompt variants and source conversations before splitting to reduce overlap. Even unlabeled text can leak evaluation structure into a quantizer when its statistics or reconstruction errors guide the chosen representation. Labels are not required for this kind of selection dependence.

If a critical regime is rare, intentionally allocate calibration examples to it rather than hoping a small random sample includes it. Stratification changes the sampling distribution, so record that choice and evaluate using the actual deployment mixture or separately declared slices. Do not claim a universal minimum sample count: some ranges stabilize quickly while rare channels and long sequences need more coverage. Compare parameter stability across independent calibration subsets, and investigate large changes rather than silently averaging them away.

Store dataset identifiers or approved fingerprints, sampling seeds, preprocessing versions, tensor observer configuration and the resulting scales with the artifact. Ensure the evaluation report identifies which settings were selected using which data. Repeat measurements for deployment-relevant shapes and phase boundaries; some serving paths use separate kernels or precision policies. When deployment shifts, rerun representative calibration and held-out comparisons together. A newly observed clipping spike is evidence to investigate, not permission to retune using the final evaluation answers.

For a coverage calculation, Reset the probability lab and set both p=0.1 and n=20. Predict mean two rare examples and probability of zero approximately 0.1216. In a separate run Reset and set both p=0.1 and n=30; predict mean three and zero probability approximately 0.0424. Play highlights exact binomial outcomes rather than sampling calibration records. This iid model explains a coverage risk; it cannot certify that actual prompts are independent or that any observed range is sufficient.