Long-context release regression suite

Record model weights, tokenizer, positional settings, cache policy, serving version, prompt template, generation settings, corpus snapshot, and source mapping. A tokenizer change can move evidence or alter whether a prompt fits, so store both source text and actual token counts for each release. Include instructions, separators, and output reserve in the length condition. Warm and cold prefix-cache states are distinct performance cases. Without that metadata, an apparent model regression may actually come from different visible input, truncation, or runtime behavior.

Choose lengths spanning the intended range, evidence positions, languages, and workload families. Include exact lookup, several separated facts, summarization constraints, code dependencies, and citation checks where the product supports them. Use held-out sources and fresh synthetic values, while keeping realistic tasks alongside synthetic probes. A suite generated from the same templates used for adaptation can overstate robustness. Preserve expected answers, support spans, and uncertainty labels so a changed output can be evaluated against evidence rather than merely compared as a different string.

Compare releases on matched cases with a fixed metric definition. Keep failures such as rejection, timeout, missing citations, and incomplete outputs in their intended denominators. Report class slices and worst required conditions, because a gain on short easy inputs can hide a regression on the exact long workload being released. Decide pass criteria and tolerances before inspecting the final comparison. A quality threshold, a support audit, and a latency budget answer different questions and should not be collapsed into one unexplained score.

Use grouped uncertainty analysis for related variants of a base case, and distinguish exploratory diagnosis from final holdout evidence. When a slice regresses, inspect visibility, candidate coverage, position handling, compressed provenance, and serving state before selecting a correction. Record numeric tolerance and decoding variability; a fixed seed does not automatically guarantee bitwise-identical execution across every device and backend. Repeatedly rerunning until a favorable sample appears is not evidence that a regression disappeared. Retest the implicated condition and neighboring slices after a change.

Publish the tested length and workload envelope with the exact configuration and unresolved limitations. A successful run outside that envelope can motivate another experiment, but does not automatically expand the release claim. Keep resource traces that explain cache growth, prefill scheduling, and decode latency under representative arrivals. If a fallback changes retrieval or compression, include it in both quality and timing tests. The user experiences the complete system, so a component benchmark cannot stand in for an evidence-grounded end-to-end release comparison.

The probability lab illustrates a deliberately assumed regression-detection trigger model. Reset, set trigger probability to 0.1 and trial count to 10, predict the chance of at least one trigger, then Step: approximately 0.6513. Compare a fresh Reset with both probability at 0.1 and trials at 30 before Step: approximately 0.9576. Real suite cases are often correlated and trigger probability is unknown. The exact binomial model does not execute requests, validate citations, simulate serving load, or prove that a release has no untested regression.