A release claim needs an immutable evidence bundle and visible regressions

Record task IDs and families, initial snapshots, policy and model revisions, prompts, tool schemas and implementations, observation modes, user-simulator configuration, grader versions, resource limits, seeds where meaningful, and termination rules. Reference exact output artifacts, effect evidence, traces, and human adjudication records by stable identity. A model API label can change behind the same name, so record available version evidence and reproducibility limits. A complete manifest should also identify setup failures, missing results, and exclusions under the declared protocol instead of describing only the successful examples.

Use deterministic manifest serialization and suitable integrity digests for referenced bytes, while distinguishing integrity from trustworthy interpretation. A hash can detect a byte change under its assumptions but does not prove that a grader was correct or a trace truthful. Keep sensitive content behind appropriate access boundaries with minimal public metadata. Harness caches also need provenance: the official SWE-bench guide warns that reused run and instance identifiers can reuse earlier results. Check the pinned implementation and artifact identity before interpreting a rerun as evidence for a changed patch or policy.

Evaluate old and new systems on the same task versions and resource protocol, with repeated conditions paired where appropriate. Build the joint table: both pass, old-only pass, new-only pass, and neither. An aggregate improvement can hide regressions in important task families or monitored constraints. Preserve counts and uncertainty at the task-cluster level. A release gate can require additional conditions beyond mean success, but it should name them explicitly and show their evidence rather than quietly redefining a task's outcome after the comparison.

A newly discovered failure should become a reproducible regression case with its original full trace and a validated reduced version when useful. Once engineers tune against it, it belongs to development coverage; it should not continue to count as untouched final holdout evidence. Keep a separate evaluation set or protocol for claims about generalization. Environment and grader updates can also change old labels, so version them and rerun comparable baselines when necessary. Otherwise a report can confuse a better checker or easier fixture with an improved agent.

Publish the tested conditions, uncertainty, known regressions, unresolved cases, and evidence-access process alongside the claimed improvements. Distinguish a staged simulator result from target-system validation and a finite monitor pass from an unrestricted behavioral guarantee. Zero observed failures over a finite suite does not establish a zero failure probability, especially with clustered tasks and untested conditions. A release decision can still be justified for a specified use and operating point; its strength comes from explicit evidence and limits, not from calling the benchmark comprehensive.

The probability lab shows why a clean finite run is compatible with a nonzero assumed defect rate. Reset, explicitly set BOTH p=0.01 and n=30, predict zero observed defects with probability about 0.7397, then Play or Step. Reset and re-set BOTH p=0.1 and n=30; predict about 0.0424. Edits retain t and playback highlights exact iid counts. The toy has no release manifest, task coverage, real defect estimate, or regression gate. It illustrates one sampling fact without claiming that independent trials describe the actual task suite or that the slider measures deployment risk.