Repeated trials estimate reliability only under a stated task and sampling unit

Record policy revision, model sampling settings, tool and user-simulator versions, seed information, budgets, reset procedure, and stopping rules. Independent resets are needed to avoid carrying solved state into later attempts, but equal seeds alone do not guarantee identical remote-service or hardware behavior. If a system adapts after each failed attempt, it defines a sequential policy different from independent draws of one fixed policy. Evaluate that adaptive procedure explicitly instead of treating its dependent attempts as interchangeable samples for a formula derived under fixed conditions.

The original code-model evaluation paper uses pass@k for the probability that at least one of k sampled candidates passes its checks. The original tau-bench paper introduces an all-k consistency measure commonly written pass^k. These events move in opposite directions as k grows under a nontrivial fixed success probability. Neither notation should be silently substituted for the other. A one-of-k evaluation can use a hidden grader to recognize success, while a deployed agent may lack that oracle and must pay for its own selection or verification mechanism.

Among size-k subsets, all attempts pass only when every chosen member comes from the c successful observations. The corresponding ratio estimates a different reliability event from finding at least one pass. Under conditional iid success probability p, the target events are 1−(1−p)^k and p^k respectively. Plugging an observed c/n into those nonlinear expressions is generally different from the finite-sample unbiased estimators. Do not call the discrepancy an arithmetic error: one conditions on the observed pool, while unbiasedness concerns expectation over repeatedly sampled pools under the stated model.

Tasks usually have different difficulty. Compute the intended per-task quantities before averaging across a declared task distribution; pooling successes across unrelated tasks can answer another question. Repeated attempts within a task can be correlated through shared fixtures, user behavior, or policy state. For uncertainty about new tasks, preserve task clusters in a suitable analysis or resampling design rather than pretending every trial is an independent new task. More attempts help characterize within-task variability but do not expand coverage to unseen environments, languages, or action contracts.

Predeclare how timeouts, invalid actions, setup failures, missing outputs, and grader errors are reported. Excluding unsuccessful or expensive attempts after observing outcomes inflates reliability estimates. Report single-attempt success, repeated-trial metrics, trial counts per task, budgets, and relevant uncertainty together. If a task has fewer than k valid sampled attempts under the metric protocol, do not pretend the combinatorial estimator is defined there without a stated missing-data rule. Preserve the failure traces so variability can be attributed to actions and environment conditions rather than dismissed as model randomness.

The probability lab displays a hypothetical count distribution. Reset, explicitly set BOTH p=0.6 and n=5, predict mean three, then Play or Step. Reset and re-set BOTH p=0.6 and n=10; predict mean six. Edits retain t and playback highlights exact iid outcomes. There are no task clusters, adaptive retries, hidden graders, or pass@k calculators. Its assumed p is not inferred from the observed count, and it does not show that real repeated agent trials meet the independence or fixed-policy conditions needed by the estimators.