A successful final state does not establish every property of the execution

Choose concrete events such as tool submission, accepted authorization, committed mutation, acknowledgment, cancellation, and completion. Distinguish a requested action from a durable effect: a timed-out request may have committed, while a confident action description may never have reached a tool. Where the task requires authorization, bind its scope to the actor, target, operation, and applicable lifetime rather than checking that the word approved appeared somewhere. Prior valid authorization can remain applicable under its contract; a monitor should not invent a requirement for a new approval at every step.

Instrument the relevant boundary and preserve ordering information, including request identifiers and causal links across asynchronous services. Wall-clock order from unsynchronized hosts can be misleading. Missing or dropped events should lead to a declared unknown or incomplete result where the property cannot be assessed, rather than silently counting as compliance. An agent-written narrative is useful evidence but cannot substitute for the trusted effect stream when the property concerns actual mutations. Likewise, an effect log may omit private reasoning, so do not claim it proves a property about every internal computation.

Some properties can be falsified by one observed event, such as a committed operation outside the allowed scope. Others require a completion condition or a deadline, such as eventually reconciling a transaction. A finite unfinished trace may leave the latter unresolved. Define the monitor's termination semantics and distinguish pass, fail, and unknown when needed. A checker for one authorization invariant does not establish correctness of values, task completion, privacy, or every other constraint. Keep property definitions versioned alongside the action and environment semantics they depend on.

Do not infer process compliance from an outcome pass or infer task success from a clean process trace. The original tau-bench setting combines domain policies, API tools, and user interaction, illustrating why conversational behavior and end-state checks can be evaluated together while remaining distinct. Trace monitors can locate specific violations, and outcome graders can detect missed effects that a syntactic action checker overlooks. Both can have instrumentation and specification defects, so construct known passing, failing, and incomplete traces to validate the combined evaluation.

For every fixed cohort, report both individual pass rates and the joint categories, with unknown assessments shown separately under the reporting protocol. Inspect outcome-only passes for hidden process failures and property-only passes for incomplete goal execution. A single composite score can hide these patterns unless its conjunction and weights are explicit. Preserve the smallest decisive event evidence and the full replay context needed to understand it. Do not remove a difficult trace from the denominator merely because its monitor exposes a new instrumentation gap after results are collected.

The probability lab counts hypothetical independent property passes only. Reset, explicitly set BOTH p=0.7 and n=20, predict mean fourteen, then Play or Step. Reset and re-set BOTH p=0.8 and n=20; predict sixteen. Edits retain t; playback highlights exact binomial outcomes. There are no authorization events, causal links, outcome labels, or monitors. This toy cannot calculate joint compliance from a marginal rate; the overlap table in the example supplies that missing information explicitly rather than assuming the two checks are independent.