Define the population and quantity behind an evaluation score

An estimand is the quantity the study intends to learn, such as expected rubric success for eligible requests arriving during a specified deployment period. Fix the system identity, including retrieval corpus, tools, prompt, decoding settings, and allowed retries. A model with a larger tool budget is a different evaluated intervention. The scoring rule can be binary, continuous, or a vector of outcomes, but define the unit and direction. An arithmetic average across examples is a statistic; interpreting it as a population performance estimate requires assumptions about how those examples were obtained.

Choose whether the unit is a request, conversation, user, repository, or another entity. Averaging all turns weights long conversations more heavily than averaging per-conversation scores. Likewise, one prolific user can dominate a request-weighted score while contributing one unit to a user-weighted estimand. Neither convention is inherently correct for every decision. Record the one that matches the release question and retain identifiers that make its denominator reconstructible. If deployment changes the mix of users or tasks, the old population target may no longer describe the new decision.

A test can deliberately oversample difficult or rare tasks to obtain useful evidence. Its raw mean then represents the test mixture, not automatically the deployment mixture. For a partition into task strata, combine stratum estimates using declared target weights when that population average is the question. Weights must cover the target population and sum to one; overlapping slices cannot simply be added as though they were a partition. Uncertain or changing population weights are another uncertainty source, not a correction supplied by making the test set larger.

Holistic Evaluation of Language Models emphasizes multiple scenarios and metrics because a single aggregate hides meaningful differences. Apply that principle to a concrete release: report the chosen aggregate together with important stratum estimates, support counts, and failure consequences. A weighted success rate does not encode every deployment constraint. A critical slice may require its own minimum evidence and gate even if its population share is small. Keep such decision rules explicit rather than conceal them inside an undocumented average or an attractive leaderboard rank.

Specify what happens when generation times out, a tool fails, or an annotator cannot score an answer. Dropping those records estimates performance conditional on successful processing or labeling, which may exclude the hardest requests. Alternatively, a declared end-to-end success rule can count particular system failures as failures while separately reporting their cause. Preserve both the eligible population count and the scored denominator. A confident estimate for one old workload does not establish performance under a changed retrieval source, language mix, or user behavior.

The sampling lab isolates the simplest measurement situation. For each separate view, Reset, set both p=0.6 and n to 25 or 100, predict the variance, then inspect. Its exact Bernoulli sample-mean variances are 0.0096 and 0.0024. There is one fixed population probability, independent identical trials, and no weighting, missingness, or deployment shift. The instrument makes finite sampling variation visible; it cannot choose the appropriate estimand or turn an unrepresentative convenience test into a representative one. Those choices belong in the study protocol before results are read.