Engineering / research
Evaluate complete agent executions
Construct task environments, trace graders, property checks, and replayable failures so agent evaluations measure outcomes as well as plausible intermediate text.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Evaluate complete agent executions
Construct task environments, trace graders, property checks, and replayable failures so agent evaluations measure outcomes as well as plausible intermediate text.
- 01
An agent task is an initial state and action contract, not only a prompt
Specify environment snapshots, observations, tool effects, budgets, and reset boundaries so comparisons concern reproducible executions rather than inherited side effects.
29 min - 02
An outcome grader must test effects while preserving an independent target
Design state-based checks, protected holdouts, and grader audits that distinguish task success from convincing text, test manipulation, or incomplete evaluation coverage.
30 min - 03
A successful final state does not establish every property of the execution
Define monitored event properties, separate observed actions from durable effects, and report outcome and behavioral compliance as a joint result rather than an assumed equivalence.
29 min - 04
A simulator's plausible responses need a validated state-transition boundary
Compare tool contracts and paired outcomes, expose omitted failure modes, and distinguish simulator success from evidence of transfer to a target system.
30 min - 05
Repeated trials estimate reliability only under a stated task and sampling unit
Separate one-of-k success from all-k consistency, derive finite-sample estimators, and preserve task clusters and repeated-run configuration in uncertainty analysis.
31 min - 06
Step accuracy alone does not determine long-horizon execution reliability
Use conditional probabilities and dependence bounds, distinguish flawless paths from recovered outcomes, and test shared failure modes across extended executions.
30 min - 07
Agent comparisons need matched resource rules and complete execution cohorts
Define multidimensional budgets, count failed-run costs, and compare success-cost tradeoffs without treating a normalized scalar as a substitute for matched experiments.
30 min - 08
Minimize a reproducible failure without confusing a smaller trace with its root cause
Separate replay modes, preserve causal preconditions, use valid failure predicates, and distinguish deletion minimality from global minimality or causal explanation.
30 min - 09
Human disagreement is evidence to investigate, not a truth label to erase
Design blinded independent review, calculate agreement and chance-adjusted summaries, and adjudicate with task evidence while preserving rubric changes and uncertainty.
30 min - 10
A release claim needs an immutable evidence bundle and visible regressions
Bind policy, environment, grader, budgets, and full outcomes to a release identity, then compare paired changes without treating aggregate gains or finite clean runs as universal guarantees.
31 min