An outcome grader must test effects while preserving an independent target

Define what must change and what must remain valid. A file-edit task can require a specific behavior plus preservation of unrelated functionality; a database task can require the correct record, ownership, and committed status. Use durable state or execution checks when the goal concerns effects, rather than a regex that finds the word done in the agent's response. SWE-bench evaluates patches through repository tests, while WebArena uses functional task evaluation. These are concrete operationalizations with limits: passing their checks is evidence within those checks, not a proof of all possible application behavior.

Separate agent-controlled artifacts from grader authority. Keep expected values, protected fixtures, and grader configuration outside the agent's writable or visible scope where the protocol requires concealment. A patch that disables an assertion or changes an evaluation script should not acquire a passing result merely because it made the checker easier. Validate positive and negative cases, unrelated but plausible outputs, and partial-completion cases. Where several implementations satisfy the user goal, avoid a brittle exact-string target that rejects valid alternatives solely for stylistic differences.

Split development and final evaluation by meaningful task families or source relationships, not only by randomly shuffling near-identical prompts. Holdouts can leak through public solutions, training material, environment artifacts, or repeated feedback. Record overlap checks and exposure limitations; the absence of a detected match does not establish perfect independence. Keep hidden expected outcomes out of the agent's observation stream and avoid letting grader error messages disclose the full target during repeated tuning. Once a case is used to develop a fix, treat it as a regression case rather than pretending it remains untouched final evidence.

A state checker can still have false positives, false negatives, flaky dependencies, or insufficient coverage. Audit accepted and rejected examples, including boundary cases and tasks with several valid outcomes. If human review supplies the audit label, preserve its rubric and uncertainty rather than calling disagreement a machine-grader error by definition. Grader sensitivity and false-positive rates can vary by domain, so a pooled correction need not transfer to every task family. Report setup and evaluation failures separately under the declared cohort policy instead of removing hard cases after seeing their results.

Store the exact candidate artifact, grader revision, protected fixture identity, logs, and decisive state evidence. Distinguish malformed submissions, grader crashes, and genuine goal failures. A polished explanation can be useful for diagnosis while still having no authority over the effect predicate. Conversely, a terse answer can accompany a correct state change. For goals that require qualitative judgment, use an explicit rubric and independent review rather than assuming a model judge's fluent rationale validates its own label. Test for sensitivity to irrelevant formatting and agent identity where those should not matter.

The network lab illustrates a score threshold without validating its meaning. Reset, explicitly set BOTH x=1 and w=2, predict output about 0.8176, then Play or Step. Reset and re-set BOTH x=0.5 and w=2; predict about 0.6225. A paper threshold 0.7 accepts only the first. Edits retain t; bias is fixed at −0.5. There is no task state, hidden test, calibrated outcome classifier, or audit truth. A bounded score and threshold do not establish that the accepted execution produced the required effect.