An agent task is an initial state and action contract, not only a prompt

Identify task instances and families, then record the initial data snapshot, application and dependency versions, identities and permissions, clock assumptions, observation interface, and allowed external resources. A screenshot agent and an agent receiving structured database rows do not solve the same observation problem even when the final goal is similar. WebArena and OSWorld illustrate task-specific environment setup and execution-based checking in web and desktop settings. Their realism and reproducibility are useful properties within their declared domains, not proof that one suite represents every deployment environment.

Specify which state is reset between attempts and which shared resources remain. Files, cookies, service caches, queued jobs, search indexes, and background workers can survive a superficial process restart. Validate setup preconditions before running the policy and record setup failures separately under a predeclared reporting rule. Seeds help reproduce some random choices but do not identify changing remote services or every nondeterministic execution. Retain immutable snapshot and configuration identities rather than relying on a mutable image tag or the word latest in an evaluation report.

A tool call can fail before an effect, succeed with an acknowledged effect, or commit an effect while its response is lost. Record this ambiguity in the environment contract. An error string alone does not establish that no payment, file update, or message occurred. A retry with an enforced idempotency key can differ fundamentally from a second unkeyed action. The evaluator should inspect the authoritative effect boundary rather than count confident completion text or assume all repeated calls are harmless. Use controlled fixtures for such tests so evaluation does not perform unintended real external actions.

Observation freshness also matters. An action may update a database before a search page refreshes, or an asynchronous job may complete after the agent stops. Define the completion boundary, waiting policy, and maximum observation delay. A task can require one final state, a sequence of allowed states, or a durable artifact that survives restart. Those requirements belong in the grading specification. If the environment changes between systems, report that as a changed experimental condition instead of attributing every difference to the agent policy.

Maintain task ID, environment version, agent version, attempt ID, setup evidence, action budget, termination reason, and outcome artifacts for every scheduled execution. Do not let one attempt's completed goal make the next attempt trivially successful. Counterbalance or isolate conditions when ordering can affect service state. The SWE-bench evaluation guide illustrates why harness execution, graded outcomes, missing predictions, and infrastructure failures are different result categories. A process finishing without a crash is not the same event as an agent solving its task.

The cache lab isolates surviving state between accesses. Reset, explicitly set BOTH capacity C=4 and stride s=4, predict twelve hits at the initial sixteen requests, then Play or Step. Reset and re-set BOTH C=3 and s=4; predict zero hits. N=min(4096,16+floor(2 t)); edits retain t and recompute from empty. The toy has no task suite, files, accounts, or environment reset. Its remembered identities illustrate why prior state affects observations, while actual evaluation isolation requires inspecting and restoring the real state boundaries.