Preserve the evidence chain from eligible cases to the release decision

An evaluation manifest should identify model weights or the strongest available provider revision, tokenizer, system and task prompts, retrieval snapshot, tool configuration, decoding parameters, and resource limits. Include dataset bytes or immutable object identities, inclusion queries, slice definitions, source-group mapping, and rubric version. Record software and scorer versions, seeds or generator states where relevant, hardware and numeric settings, and the supported limits of determinism. A seed does not promise identical outputs across arbitrary kernels, provider changes, or model revisions.

Preserve generated responses, tool results, raw judgments, parse outcomes, per-case scores, and eligibility or exclusion reasons under appropriate access restrictions. Pinned LM Evaluation Harness v0.4.9 documents model and task configuration plus sample logging; those options are building blocks rather than a complete guarantee for every custom evaluation path. Check what your runner actually records. If a remote provider cannot supply immutable model identity or determinism, record that limitation honestly and timestamp the run instead of inventing a reproducibility guarantee.

A digest binds a recorded identifier to bytes under a specified algorithm and verification process. It does not prove that the labels are correct, the sample is representative, or the system meets its release requirements. Hash the actual artifacts or a documented canonical representation and retain the referenced content. RFC 8785 defines one constrained JSON canonicalization scheme; arbitrary serializer output and semantic equivalence are not automatically the same bytes. Mutable labels can point to immutable releases, but the report must record the resolved identity used for that run.

Create a required-evidence inventory and verify the referenced objects, schemas, signatures where used, and cross-links. Check that per-case outcomes reconcile with aggregates, that every eligible record has its declared disposition, and that the uncertainty procedure used the intended grouping. Evidence completeness is only one audit dimension. A fully present but contaminated test or invalid statistic is still invalid. Conversely, a missing raw-output artifact may prevent rescoring even when the original aggregate was computed correctly. Record what is verified, unavailable, or only attested by another service.

A scoring bug or rubric update should create a new versioned result linked to the preserved original. State which outputs were rescored, which data and model identities stayed fixed, and how eligibility or labels changed. Recompute the affected metrics, uncertainty, and release gates rather than only edit the headline number. If the fix was chosen after inspecting a confirmatory set, disclose that adaptation and determine whether independent confirmation is needed. Preserve failed runs and negative findings so an auditor can understand selection rather than see only the winning report.

The probability lab illustrates a hypothetical completeness requirement without auditing files. Reset, set both p=0.9 and n=10, predict the all-ten-success bar, then inspect. It is 0.9^10≈0.34868 if ten required items are independently available with the same stipulated probability. Real artifacts can share a storage failure and are not verified by this assumption. The display has no manifest, hashing, access control, or semantic validation. Its narrow lesson is that high per-item availability does not imply high all-item availability; actual evidence must be retained and checked against the concrete inventory.