A simulator's plausible responses need a validated state-transition boundary

Inventory schemas, validation rules, state changes, permissions, idempotency, asynchronous completion, retries, rate limits, and errors. A generated response that says an order shipped should not count as a state transition unless the simulator updates the corresponding authoritative record under its contract. User simulation is another boundary: a language model playing a customer can disclose information, misunderstand instructions, or cooperate differently from actual users. The original tau-bench combines simulated user interactions, domain tools, and policies; these components should be versioned and evaluated separately rather than treated as interchangeable sources of realism.

Specify which conditions are intentionally omitted. A deterministic simulator can be useful for regression while lacking latency variation or concurrent writers. WebArena's functional sites and OSWorld's computer environments increase fidelity for particular interfaces, yet their task and application versions still bound the evidence. Higher visual detail is not necessarily higher fidelity for database commitment or authorization semantics. Choose the boundary according to the capability claim, and avoid calling a benchmark realistic as a substitute for listing which real behaviors it reproduces.

Build controlled conformance cases for invalid arguments, stale reads, committed effects with lost responses, duplicate retries, permission changes, partial failures, and cancellation. Compare both visible outputs and authoritative effects. Run these checks in isolated target fixtures, since replaying arbitrary evaluation actions against production can create new effects rather than merely observe fidelity. Pair task conditions and preserve the initial state for each comparison. If the real service has nondeterministic timing, repeated target trials may be needed to characterize the distribution instead of matching one convenient response exactly.

Outcome agreement alone can conceal divergent intermediate states that later actions expose. Conversely, harmless response formatting differences need not invalidate a task whose parser and action contract permit them. Classify mismatches by mechanism and measure whether the agent relies on the omitted behavior. A simulator patch can change task difficulty or the user policy's behavior, so a new version requires renewed comparisons. Do not calibrate only on successful demonstrations and then claim fidelity for error recovery, because that excludes precisely the transitions where many agents fail.

Keep the simulator revision, target revision, paired task list, conformance cases, tool-response samples, and state evidence with the report. Separate simulated evaluation from target validation rather than pooling both denominators into one attractive success number. Report uncertainty and task-family coverage. When the simulator improves after a discovered mismatch, add a regression case but retain a separate untouched transfer check. A finite set of matching executions supports a scoped empirical claim, not universal agent capability, complete policy compliance, or immunity to future interface changes.

The Markov lab isolates sensitivity to a transition model. Reset, explicitly set BOTH p=0.1 and q=0.2, predict state-one probability about 0.2533 after four transitions, then Play or Step four times to t=2. Reset and re-set BOTH p=0.3 and q=0.2; predict 0.5625. Both runs start in state zero; n=min(2000000,floor(2 t)), and edits retain t. This exact two-state discrete chain has no tools, simulator validation, or task policy. Its changed occupancy shows that altered transition assumptions can matter even when state labels and output format stay the same.