Measure what adaptation leaves behind

Choose held-out tasks relevant to existing use, including factual or domain checks, reasoning formats, languages, tool schemas, and behavior constraints where appropriate. Keep their source groups separate from training and from the validation used to tune mixtures. A narrow retention set cannot support a claim that nothing was forgotten globally. Record the base checkpoint's performance before adaptation under the same tokenizer, template, and evaluation procedure so a later difference is not caused by formatting or scorer drift.

Adapters may leave original base bytes unchanged while altering logits, representations, and generated responses whenever enabled. Disabling a pure adapter can recover a base execution path only if other trained modules or artifacts did not also change and the serving stack is identical. That is a rollback mechanism to verify, not proof that the adapted model preserves capabilities. Empirical forgetting studies motivate measurement but do not establish a universal rank, learning rate, or mixture that prevents regressions.

For binary correctness, each paired difference is−1,0, or 1. The uncertainty of the mean depends on those differences, not two unrelated binomial standard errors. A simple independent-item estimate uses sample variance of differences divided by item count. Related prompts, conversation variants, and task clusters violate that independence assumption;resample or model the appropriate group and report the effective evaluation unit. Repeated stochastic generations also need a design that distinguishes item uncertainty from decoding variability.

A normal approximation is illustrative for a sufficiently suitable sample, not an exact confidence guarantee. Bootstrap intervals likewise inherit their resampling assumptions and can be unstable with few groups. Report effect size and per-task results, not only whether an interval crosses zero. A statistically clear regression may be unacceptable even if a much larger target-task gain raises a pooled average. Conversely, a tiny noisy change should not be described as certain forgetting from one seed.

Use predefined acceptance rules for target improvement, retention floors, severe failure rates, and latency or resource limits. Inspect regressions by source group and task rather than compensating every loss with more wins on an easy benchmark. Reevaluate after merge, quantization, and serving changes because the final artifact can behave differently from the training-time adapter. Preserve failures as reviewed evaluation cases without repeatedly training on the final test and still claiming it remained untouched.

The probability lab contrasts expected marginal scores with a paired comparison. Reset and set both p=0.8, n=20, predict sixteen successes. Reset and re-set p=0.6, n=20 to predict twelve. These independent binomial models do not encode which items changed from correct to incorrect or the covariance of paired outcomes. Play highlights possible counts, not model answers. Explain why the same two marginal accuracies can correspond to different paired uncertainty and why the actual per-item results must be retained.