Release a versioned behavior change with reviewable evidence

A candidate includes weights and adapters, tokenizer, chat template, prompt construction, decoding settings, tool schema, output parser, and any filtering that affects observed behavior. Version the baseline under the same relevant contract. A changed sampling temperature can alter preference outcomes even when weights are unchanged. Retain the exact evaluation prompts and allowed context, judge or annotator protocol, display randomization, tie handling, and inclusion criteria. Ensure the final assessment has not repeatedly selected the same candidate during development; otherwise reserve fresh confirmation evidence or qualify the selection-conditioned result.

Choose evaluation slices based on the intended deployment: task correctness, instruction adherence, language coverage, formatting, or other concrete requirements. Blind model identities when assessing content and inspect length and position effects rather than assuming one judge removes them. Benchmarks such as RewardBench 2 provide defined evidence about reward-model comparisons, while policy release evaluation asks how the actual candidate behaves under its execution configuration. Report those purposes separately. An improved reward benchmark does not alone establish better deployed policy behavior, and a preferred policy does not automatically make its implicit reward universally calibrated.

Use a paired or grouped analysis when the same prompts, annotators, or repeated judge runs contribute several observations. A narrow interval from treating all related votes as independent can overstate confidence. Track task and population slices alongside the aggregate and state which thresholds were chosen before inspecting the candidate. If a requirement fails, diagnose it rather than averaging it away with unrelated successful comparisons. Different applications can choose different tradeoffs, but the published decision should reflect the declared contract and the population actually assessed.

Keep preference judgments and factual or executable task checks as distinct evidence. A fluent wrong answer can win a style comparison; a correct answer can lose under an unsuitable rubric. Investigate disagreement between these signals and inspect examples. Compare proxy reward trends with held-out outcomes over relevant optimization checkpoints, remembering that repeated checkpoint selection consumes development evidence. Unmeasured rare failures and distribution shifts remain outside a finite evaluation's guarantee. State those limits concretely through the tested prompt population and requirements instead of calling one score a complete safety or utility certificate.

Retain candidate and baseline identities, data and rubric versions, exact evaluation settings, uncertainty method, slice results, inspected failure examples, and the decision's scope. Bind the deployed configuration to the evaluated artifact rather than a mutable model alias. For a staged rollout, predefine observable regression triggers and who evaluates them under the authorized process. A rollback target includes the prior weights and compatible tokenizer, prompts, tools, parser, and service configuration; returning only the old weight file may not reproduce its former behavior. Exercise restoration before relying on it during an incident.

The sampling lab illustrates how independent binary evidence can vary. For each separate view, Reset, set both p=0.6 and n to 25 or 100, predict the sample-mean variance, then inspect. Variances are 0.0096 and 0.0024 under the independent common-p model. The bars are exact probabilities, not simulated evaluation votes, and the model has no ties, prompt clusters, task checks, or release gate. It shows one reason to consider evidence size, while real uncertainty must match the actual design and cannot establish untested deployment requirements.