Make the release rule explicit across metrics, slices, and uncertainty

List the metrics and slices that matter to the deployment decision. State whether larger or smaller is better, its unit, population, required support, and threshold. A quality proportion, an unsafe-action rate, a percentile latency, and a cost measure cannot be added without an explicit value model. A conjunctive rule can instead require each critical condition separately. Preserve a declared action for missing or inconclusive evidence, such as withholding release or limiting deployment scope. Changing a threshold after seeing the result is a policy revision that needs a new evidential accounting.

Use uncertainty summaries appropriate to each metric and design. A lower bound can support a minimum-quality requirement, while an upper bound can support a maximum-error or latency requirement, provided the bound procedure actually has the stated coverage. Percentiles and clustered outcomes need methods suited to their sampling structure. A lower mean latency does not imply a lower tail percentile. A precise aggregate cannot compensate for a critical unassessed slice unless the release policy explicitly permits that tradeoff and records who accepted it.

If the report claims all of several confidence statements simultaneously, marginal confidence levels alone do not guarantee the same joint level. A union bound can allocate a total error budget across valid component procedures without assuming independence. This can be conservative when metrics are related. Holm's sequentially rejective procedure is a classic alternative for a specified family of hypothesis tests; applying it requires valid component p-values and its actual testing rule. Confidence intervals, null-hypothesis tests, and an operational release checklist are different objects and should not be relabeled interchangeably.

A conjunction that releases only when every required test passes has its own intersection-union testing logic. Do not claim that Bonferroni is universally necessary for such a gate, or that an independent toy probability of any error equals the gate's false-release probability. State whether the desired guarantee concerns simultaneous reported bounds, multiple discoveries, or a particular composite release null. Select the method for that claim before inspecting results. Multiplicity from many candidate models, slice searches, or threshold choices also belongs in the design rather than only counting the four final displayed metrics.

A statistical error allocation limits specified inferential mistakes under a model. An operational error budget may instead limit failed requests or service-level violations over a period. Neither quantity determines the other without an explicit decision model. Freeze sample sizes and comparison rules for fixed-sample claims. Repeated dashboard looks and stopping on a favorable result require a justified sequential design; Howard and colleagues develop time-uniform confidence sequences under stated assumptions. Such methods still do not erase selection from unmodeled feedback or guarantee coverage after an arbitrary population shift.

The probability lab illustrates independent marginal error accumulation only. Reset, set both p=0.05 and n=4, predict the probability of at least one error, then inspect one minus the zero-success bar. It is 1−0.95^4≈0.18549, compared with the union-bound upper limit 0.20 for four 5% events. Actual metric errors need not be independent, and these events are not the release decision itself. The instrument has no confidence interval construction, gate policy, or sequential monitor. Use it to distinguish a toy joint-event probability from the real guarantee requested by the evaluation protocol.