Construct a test that preserves the intended separation and sampling design

The sampling frame is the inventory from which cases can actually be selected. A log export may omit unauthenticated users, failed requests, certain languages, or deleted sessions. Random selection from that export is random relative to the frame, not automatically representative of all deployment traffic. Record eligibility, collection dates, inclusion probabilities where known, and the exclusions. Separate a probability sample intended for population estimation from a curated challenge suite intended to expose mechanisms. Both can be useful, but their raw scores support different generalization claims.

For a finite frame with known inclusion probabilities, design-based weighting can estimate the mean of fixed outcomes under that sampling design. Every target item needs positive inclusion probability, and the estimator and variance method must match the design. A convenience sample with unknown selection is not repaired by assigning arbitrary weights afterward. In a stratified design, retain stratum sizes and selection counts; in a clustered design, retain cluster IDs. The information discarded during dataset assembly often determines which uncertainty calculations are defensible later.

If the intended claim concerns unseen conversations, keep all messages and derived artifacts from one conversation together. For new repositories, splitting individual functions can leak shared code, templates, or near duplicates across files. Fit preprocessing, retrieval indexes, example selection, and calibration only with the information permitted by the protocol. A future-oriented test also needs point-in-time feature and source cutoffs. A random row split cannot prevent a later document or label from revealing an answer that was unavailable when the target decision would have occurred.

Train, development, and final-test roles should be stated separately. Development examples may guide prompt or threshold choices, while the confirmatory set is reserved for the declared final comparison. Repeatedly examining final scores to choose a model makes that final set part of selection, even if no gradient update reads its labels. Dwork and colleagues analyze this adaptive reuse problem; their specialized mechanisms do not mean an ordinary repeatedly queried holdout remains automatically valid. Benchmark-specific splits and shot conventions also matter: preserve the original MMLU protocol when claiming that benchmark result.

Specify deduplication, normalization, answer extraction, filtering, and scoreable-outcome rules before reading the confirmatory results. Keep counts and reasons when records are excluded, including records that fail generation or parsing. A changed filter can change the task population rather than improve the model. Version the inclusion query and source snapshot so another evaluator can reconstruct the denominator. A fresh test can reduce feedback leakage but still miss the deployment population; sampling validity, access control, and temporal realism are distinct requirements that need their own evidence.

The probability lab illustrates only a simple coverage calculation for independent draws. Reset, set both p=0.1 and n=10, predict the chance of zero designated-category records, then inspect the zero-success bar. It is 0.9^10≈0.34868. The model neither builds a dataset nor detects duplicates. It shows that a ten-item random sample can miss a ten-percent category, even under an ideal known frame. Actual test construction should therefore retain category counts and design evidence instead of treating random selection as a guarantee of comprehensive coverage.