EXPLAIN plans and estimation error
Read estimates, observed rows, loops, buffers, and timing in the correct scope before testing a plan-regression hypothesis.
On this page
The question for today
What you’ll understand
- Separate planned rows and abstract cost from observed execution measurements.
- Use loops to interpret repeated-node row and time averages.
- Locate an early estimation error and design a controlled follow-up experiment.
An execution plan is evidence only when its fields are interpreted correctly. EXPLAIN describes a selected plan, while EXPLAIN ANALYZE runs the statement and adds observations. Estimated cardinalities, per-invocation row averages, inclusive timings, and buffer events answer different questions. Combining them carelessly can produce a confident diagnosis of work that never occurred or hide a repeated operation that dominates the query.
Read the estimated plan before executing it
Identify access paths, join inputs, conditions, blocking operators, order requirements, and expected row widths. Estimated rows describe what a node is expected to emit under the selected plan, not every row it necessarily examines. An index condition and a residual filter act at different stages. Startup and total costs use the planner's relative scale. Preserve parameter values, relevant settings, statistics state, and data context when saving a plan; a different bind value or data distribution can legitimately produce a different choice.
EXPLAIN without ANALYZE does not execute the requested data-processing plan, although parsing, rewriting, and planning still do work. ANALYZE executes it, including modifications when the statement writes. Use a suitable test environment and transaction strategy for such experiments, remembering that external side effects and some nontransactional effects are not erased by a rollback. A report should say which statement actually ran and whether it consumed the full result. A LIMIT or early stop can leave descendants only partly exercised.
Loops and inclusive timing prevent naïve addition
For a repeatedly executed node, PostgreSQL displays observed rows and node times as per-execution averages with a loops count. Multiplying a row average by loops estimates total emitted rows over those executions, subject to display rounding. Compare the estimated rows with the actual per-execution average before multiplying only one side. The outer input to a nested loop can make a tiny inner operation run many times, so inspect both local cost and repetition. Parallel workers and partially executed branches add further scope information that should remain visible in the saved plan.
Node timing includes work performed through its children, so adding parent and child totals double-counts overlapping execution. Multiplying an average node time by loops estimates accumulated inclusive time for that node's executions; it is not generally additive wall time across all nodes or workers. Buffer hits count database buffer events, not CPU-cache hits or a guaranteed number of distinct pages. A shared-buffer read may still be served by an operating-system cache. Separate resource-event counts from elapsed storage latency rather than converting one into the other with an unexplained constant.
Find the first consequential mismatch
Walk from inputs upward and identify where estimated and observed rows first diverge substantially. A later wide join may merely amplify an earlier filter error. Then connect that discrepancy to an algorithm choice, repetition count, memory fit, or ordering cost. Use several representative parameters and compare controlled changes such as refreshed relevant statistics, a supported extended statistic, or a narrower predicate. Disabling an operator can be a diagnostic experiment, but one faster forced plan does not by itself establish a good global default for every query.
Instrumentation itself costs work. PostgreSQL's TIMING OFF option can reduce per-node clock-reading overhead while retaining useful row observations and overall execution timing. Repeat comparisons under recorded cache and concurrency conditions. For a known-counter exercise, Reset the cache lab, set both C=4 and s=4, predict twelve hits among its initial sixteen requests and thirteen after one Step, then Step. Separately Reset, set both C=3 and s=4, predict zero hits among sixteen and zero after the seventeenth request, then Step. This exact toy LRU trace teaches count denominators, not SQL instrumentation or a processor benchmark.
Work it through
Read a repeated lookup in the same scope
A hypothetical inner lookup shows estimated rows=2 and observed average rows=5 with loops=1,000. Its reported average inclusive total time is 0.03 ms per loop. Ignore display rounding for this exercise.
Compare per-invocation row counts: actual five divided by estimated two gives a 2.5-fold underestimate. Multiplying only the actual side by loops would mix scopes.
Observed total emitted rows across those invocations are 5×1,000=5,000. If the same two-row estimate were applied to all 1,000 invocations, its corresponding total would be 2,000.
Accumulated inclusive node time is 0.03×1,000=30 ms. Do not add that blindly to a parent's inclusive time or equate sums across parallel workers with the statement's wall-clock execution time.
Rows, repetition, and time become useful together only after their invocation scope and inclusiveness are explicit.
The living notebook
Explore the model
Interactive lab · cache
Check the denominator behind a counter
Reset, set both C=4 and s=4, predict twelve hits among the initial sixteen requests and thirteen after one Step, then Step. Separately Reset, set both C=3 and s=4, predict zero of sixteen and zero after the seventeenth request, then Step. This exact LRU trace teaches counting scope, not EXPLAIN or hardware cache events.
- Hits / requests
- 0 / 16
- Hit rate
- 0%
- Address-stream period
- 16 addresses
A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA count and its denominator must describe the same scope before a ratio or comparison has meaning.
Pause & reason
Check your understanding
Your turn
Try it yourself
A scan estimates 400 emitted rows and observes 100 per execution. Calculate positive-count q-error. What should you report if it instead observes zero?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use the larger of actual/estimate and estimate/actual only when both are positive.
Compare with the solution
For actual 100 and estimate 400, q=max(100/400,400/100)=4. If actual is zero, that expression is undefined; report the zero-versus-400 discrepancy and an absolute error of 400 rows, or disclose an explicit smoothing convention. Silently substituting one can conceal a qualitatively different empty-result case.
Keep the idea
Take it with you
- Compare estimated and actual rows within the same invocation scope.
- Node timings are inclusive and buffer events are not direct device-latency measurements.
- Locate consequential early errors and control instrumentation, parameters, and workload conditions.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.