Release quantization only after held-out task regression analysis

Compare the full-precision baseline and quantized candidate using the same preprocessing, prompts, decoding configuration and evaluation rules. Pin base weights, tokenizer, template, quantizer settings, runtime and selected kernels. Hold unrelated optimizations fixed when attributing a difference to quantization. For stochastic generation, distinguish input-sampling uncertainty from decoding-seed variability and use a declared replication design. A deterministic evaluation is reproducible for its fixed inputs, but it does not eliminate uncertainty about future workloads or cover every possible output path.

Report slices chosen for product importance: long context, specific languages, numerical reasoning, structured tool arguments or rare critical workflows. Aggregate accuracy weights tasks by their sample counts, which may differ sharply from deployment importance. A large easy slice can mask a severe smaller regression. Preserve both deployment-weighted and critical-slice views, and keep memory, prefill latency and decode latency separate. A byte saving is a valid result even when quality fails a release requirement; it does not waive that requirement.

For binary outcomes, a candidate-only success contributes plus one and a baseline-only success contributes minus one. Agreements contribute zero, whether both succeed or both fail. The mean of these differences estimates the paired accuracy change. Its sample variance depends on the discordant cases, so two systems with identical headline accuracies can have very different paired uncertainty. Store per-example outcomes or approved aggregate pair counts rather than retaining only separate success totals.

The usual standard error from independent examples is a useful approximate calculation when that sampling assumption is plausible. Multiple questions from the same document or repeated variants can violate it; then resample or analyze at an appropriate independent group level. A normal interval can be poor for small samples or few disagreements. Multiple selected slices also require a predeclared uncertainty policy rather than interpreting every nominal interval as a simultaneous guarantee. An interval containing zero is not proof that the systems are equivalent.

Choose whether release requires a point estimate, a conservative interval bound or a more formal noninferiority test against a specified margin. Keep final evaluation separate from calibration and repeated configuration selection. If an important slice fails, investigate clipping, scales and kernel agreement using development data, then evaluate a newly versioned candidate under the declared process. Preserve the last accepted artifact and its complete numerical interpretation. Rollback must restore compatible weights, scales, layouts and runtime behavior, not merely a filename referring to the latest compressed checkpoint.

For a simple uncertainty analogy, Reset the sampling lab and set both p=0.5 and n=100. Predict sample-mean variance 0.0025 and standard deviation 0.05. Separately Reset and set both p=0.5 and n=25; predict variance 0.01 and standard deviation 0.1. Play highlights exact binomial outcomes rather than drawing evaluation samples. This independent Bernoulli model illustrates sample-size scaling; it does not model paired three-valued differences, correlated tasks or the release decision itself.