research35 min studyLesson 900 in Engineering

Evaluate memory and planning with matched tasks, actual cost, and failure-specific controls

Separate the effect of stored information, retrieval, search, verification, and extra compute while testing stale, misleading, duplicate, and adversarial evidence paths.

Before you begin

Stop on verified task conditions or explicit limits, not the appearance of a finished trace

On this page

The question for today

What you’ll understand

Compute paired task wins and losses alongside aggregate success rates.
Account for actual cost without interpreting extra compute as a pure memory effect.
Design ablations that preserve task lineage, validators, and failure metrics.

A memory-enabled agent can outperform a baseline because it retrieved useful evidence, because it received more tokens or tool calls, or because the test repeated information already present in its stored episodes. A planning controller can also improve an oracle best-of-many result without improving the policy that actually chooses and executes one answer. Evaluation must separate these mechanisms. Use fixed task and artifact versions, independent outcome checks, controlled resource accounts, and failure-specific tests before attributing a gain to memory or search itself.

Change one mechanism while preserving the comparison contract

Compare no persistent memory, relevant raw episodes, summaries with provenance, and retrieval-disabled or shuffled-memory controls. Match model, task instructions, tools, allowed budget, and final verification where the scientific question requires them. Record actual consumed cost as well as the allowance, since retrieval and summaries change prompt length. A random-memory control can reveal distraction or generic extra context effects, but it does not replace a relevant-data comparison. For search, vary branching, evaluator, and verifier separately rather than silently changing every component at once.

Prevent leakage through related tasks, source assets, solved examples, and stored outcomes. Split at the unit needed by the intended generalization claim, and snapshot the memory corpus before each evaluation protocol. An online-learning experiment can intentionally accumulate feedback, but then sequence and previous trials are part of the treatment. A supposedly independent test must not retrieve its own future answer or an unmarked solved duplicate. Preserve provenance so a surprising success can be traced to evidence actually available at that point rather than explained after the fact.

\widehat\Delta=\frac1N\sum_{i=1}^{N}\left(Y_{i,\rm memory}-Y_{i,\rm base}\right)

Y is the same externally defined binary task-success outcome on paired task instances, and N is positive. The difference is dimensionless and descriptive unless the experimental design supports a causal interpretation. Related tasks, stochastic runs, carryover memory, and selection affect uncertainty and must be handled at the appropriate experimental unit.↔ Scroll wide equations horizontally.

Measure failures that a success average can hide

Include stale source versions, revoked records still in an index, contradictory updates, missing provenance, exact duplicate deliveries, and tool-output instructions outside the authorized task. Test a partial effect followed by a lost acknowledgment and a restart. Measure unsupported success claims, duplicate effects, inappropriate memory promotion, and recovery cost separately from task pass rate. AgentDojo supplies one empirical attack-evaluation setting, while Reflexion and Tree of Thoughts study other mechanisms and tasks. Their results do not imply that every combined memory-and-planning system inherits all reported benefits or solves these failure modes.

Inspect paired outcomes rather than only pooled means. A method can gain on one task family and lose on another, with identical net success. Report relevant cohorts, actual cost, and uncertainty at the level of independent tasks or runs; many related tool calls do not create equally many independent trials. Do not relabel a self-evaluator score as ground truth. A correct final artifact, environment postcondition, or carefully designed outcome assessment must establish the metric being compared. Keep partial results and timeouts in a declared denominator rather than discarding difficult attempts.

c_{\rm observed\ gain}=\frac{C_{\rm memory}-C_{\rm base}}{S_{\rm memory}-S_{\rm base}}\quad\text{when }S_{\rm memory}>S_{\rm base}

C uses a defined total resource unit on the same task set, and S counts successful tasks. The ratio is a descriptive extra cost per additional observed success, not a causal marginal cost, universal efficiency score, or monetary estimate. A zero or negative success difference needs a different report rather than division by an unsuitable denominator.↔ Scroll wide equations horizontally.

Use independent replays to test the proposed explanation

Save task inputs, memory snapshots, retrieval records, plan choices, tool versions, budgets, and validator evidence for replay. If a claimed memory gain disappears when extra compute is controlled, report that mechanism rather than defending the original label. If a summary helps ordinary tasks but harms a version-sensitive cohort, inspect which qualifier or provenance was lost. A useful ablation produces a falsifiable explanation and a bounded scope for the result. It does not establish guaranteed reasoning correctness, unlimited transfer, or safe effect execution outside the tested protocol.

The probability lab shows a separate independent success-count model. Reset, set both p=0.6 and n=20, predict mean twelve and variance 4.8, then Play or Step. Reset and set both p=0.7 and n=20 for mean fourteen and variance 4.2. Exact binomial bars assume independent equal-probability trials; playback highlights outcomes rather than sampling. There are no paired tasks, memory snapshots, carryover, costs, or correlated runs. Use an experiment-specific uncertainty method rather than treating this toy law as a significance test for an agent ablation.

Work it through

More successes can require more cost and include regressions

On eighty paired tasks, baseline passes 48 and a memory configuration passes 56. There are twelve memory-only wins and four baseline-only wins; other outcomes agree. Actual mean token cost is 1200 per baseline task and 1800 per memory task. These are descriptive observations, not a claim of matched consumed compute.

Success rates are 48/80=60% and 56/80=70%, a 10-percentage-point increase. The net eight additional successes equal twelve wins minus four losses, so the aggregate hides real regressions on four tasks.
Consistent paired counts include 44 both-pass tasks and 20 both-fail tasks:44+12+4+20=80. Their pattern and task cohorts matter when diagnosing where memory helps or harms.
Total token costs are 96000 and 144000, an extra 48000. Dividing by eight additional observed successes gives 6000 extra tokens per observed gain. A pure memory-effect claim still needs appropriate compute controls and an uncertainty design.

The result

Report the effect, discordant outcomes, cost, and failure behavior together. Neither the ten-point aggregate gain nor the descriptive cost ratio proves general reasoning correctness or causal benefit from memory alone.

The living notebook

Explore the model

Interactive lab

Interactive lab · probability

An independent success law is not a paired ablation analysis

Reset, set both p=0.6 and n=20, predict mean 12 and variance 4.8, then Play or Step. Reset and re-enter both p=0.7 and n=20 for mean 14 and variance 4.2. Exact binomial masses assume independent equal-probability trials; playback highlights outcomes, not samples. The model has no paired tasks, carryover memory, resource cost, validator, or significance procedure, and n caps at 30.

Exact probabilityHighlighted outcomeMean np

Mean · np: 5
Variance · np(1−p): 2.5
Total probability: 1

Success probability p0.5Trials n10

t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forDesign the ablation and uncertainty account for the actual tasks, memory state, and consumed resources.

Pause & reason

Check your understanding

Interactive checks need JavaScript. You can still read the explanation.

Read the checkpoint explanation

Extra computation and changed selection can confound a memory claim. A mechanism-specific comparison and honest cost/failure account are needed to support the intended conclusion.

Your turn

Try it yourself

Forty tasks yield 20 baseline passes and 24 memory passes. Mean token costs are 1000 and 1500. Compute the success-rate difference, total extra tokens, and descriptive extra tokens per additional observed success. What remains unestablished?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use the same forty-task denominator and distinguish a cost ratio from an identified treatment effect.

Compare with the solution

Rates are 50% and 60%, differing by 10 percentage points. Total costs are 40000 and 60000, so the extra 20000 tokens divided by four additional observed successes gives 5000 tokens per observed gain. Causal memory benefit, uncertainty, cohort regressions, unsupported completion claims, and robustness to failure cases still need the designed controls and measurements.

Keep the idea

Take it with you

Ablate memory, retrieval, search, evaluator, and extra compute under an explicit comparison contract.
Use paired outcomes, provenance-aware splits, and failure-specific metrics alongside success.
Preserve replay evidence and describe empirical gains within their tested scope.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.