advanced24 min studyLesson 290 in Engineering

EXPLAIN plans and estimation error

Read estimates, observed rows, loops, buffers, and timing in the correct scope before testing a plan-regression hypothesis.

On this page

What you’ll understand

  • Separate planned rows and abstract cost from observed execution measurements.
  • Use loops to interpret repeated-node row and time averages.
  • Locate an early estimation error and design a controlled follow-up experiment.

An execution plan is evidence only when its fields are interpreted correctly. EXPLAIN describes a selected plan, while EXPLAIN ANALYZE runs the statement and adds observations. Estimated cardinalities, per-invocation row averages, inclusive timings, and buffer events answer different questions. Combining them carelessly can produce a confident diagnosis of work that never occurred or hide a repeated operation that dominates the query.

Read the estimated plan before executing it

Identify access paths, join inputs, conditions, blocking operators, order requirements, and expected row widths. Estimated rows describe what a node is expected to emit under the selected plan, not every row it necessarily examines. An index condition and a residual filter act at different stages. Startup and total costs use the planner's relative scale. Preserve parameter values, relevant settings, statistics state, and data context when saving a plan; a different bind value or data distribution can legitimately produce a different choice.

EXPLAIN without ANALYZE does not execute the requested data-processing plan, although parsing, rewriting, and planning still do work. ANALYZE executes it, including modifications when the statement writes. Use a suitable test environment and transaction strategy for such experiments, remembering that external side effects and some nontransactional effects are not erased by a rollback. A report should say which statement actually ran and whether it consumed the full result. A LIMIT or early stop can leave descendants only partly exercised.

q=max⁡ ⁣(AE,EA),A>0, E>0q=\max\!\left(\frac{A}{E},\frac{E}{A}\right),\qquad A>0,\ E>0
A and E are actual and estimated rows for the same invocation scope. q is a dimensionless symmetric multiplicative discrepancy, equal to one for agreement. Zero actual or estimated counts need separate absolute-error or explicitly chosen smoothing rules; this expression is undefined there.↔ Scroll wide equations horizontally.

Loops and inclusive timing prevent naïve addition

For a repeatedly executed node, PostgreSQL displays observed rows and node times as per-execution averages with a loops count. Multiplying a row average by loops estimates total emitted rows over those executions, subject to display rounding. Compare the estimated rows with the actual per-execution average before multiplying only one side. The outer input to a nested loop can make a tiny inner operation run many times, so inspect both local cost and repetition. Parallel workers and partially executed branches add further scope information that should remain visible in the saved plan.

Node timing includes work performed through its children, so adding parent and child totals double-counts overlapping execution. Multiplying an average node time by loops estimates accumulated inclusive time for that node's executions; it is not generally additive wall time across all nodes or workers. Buffer hits count database buffer events, not CPU-cache hits or a guaranteed number of distinct pages. A shared-buffer read may still be served by an operating-system cache. Separate resource-event counts from elapsed storage latency rather than converting one into the other with an unexplained constant.

Nemitted≈Lrˉ,Tinclusive≈LtˉN_{\rm emitted}\approx L\bar r,\qquad T_{\rm inclusive}\approx L\bar t
L is loops, r̄ is observed emitted rows per invocation, and t̄ is reported inclusive time per invocation in milliseconds. Products inherit rounding and execution-scope limitations. Inclusive times across nested nodes or parallel workers must not simply be summed to obtain wall time.↔ Scroll wide equations horizontally.

Find the first consequential mismatch

Walk from inputs upward and identify where estimated and observed rows first diverge substantially. A later wide join may merely amplify an earlier filter error. Then connect that discrepancy to an algorithm choice, repetition count, memory fit, or ordering cost. Use several representative parameters and compare controlled changes such as refreshed relevant statistics, a supported extended statistic, or a narrower predicate. Disabling an operator can be a diagnostic experiment, but one faster forced plan does not by itself establish a good global default for every query.

Instrumentation itself costs work. PostgreSQL's TIMING OFF option can reduce per-node clock-reading overhead while retaining useful row observations and overall execution timing. Repeat comparisons under recorded cache and concurrency conditions. For a known-counter exercise, Reset the cache lab, set both C=4 and s=4, predict twelve hits among its initial sixteen requests and thirteen after one Step, then Step. Separately Reset, set both C=3 and s=4, predict zero hits among sixteen and zero after the seventeenth request, then Step. This exact toy LRU trace teaches count denominators, not SQL instrumentation or a processor benchmark.

Read a repeated lookup in the same scope

A hypothetical inner lookup shows estimated rows=2 and observed average rows=5 with loops=1,000. Its reported average inclusive total time is 0.03 ms per loop. Ignore display rounding for this exercise.

  1. Compare per-invocation row counts: actual five divided by estimated two gives a 2.5-fold underestimate. Multiplying only the actual side by loops would mix scopes.

  2. Observed total emitted rows across those invocations are 5×1,000=5,000. If the same two-row estimate were applied to all 1,000 invocations, its corresponding total would be 2,000.

  3. Accumulated inclusive node time is 0.03×1,000=30 ms. Do not add that blindly to a parent's inclusive time or equate sums across parallel workers with the statement's wall-clock execution time.

The result

Rows, repetition, and time become useful together only after their invocation scope and inclusiveness are explicit.

Explore the model

Interactive lab

Interactive lab · cache

Check the denominator behind a counter

Reset, set both C=4 and s=4, predict twelve hits among the initial sixteen requests and thirteen after one Step, then Step. Separately Reset, set both C=3 and s=4, predict zero of sixteen and zero after the seventeenth request, then Step. This exact LRU trace teaches counting scope, not EXPLAIN or hardware cache events.

Check the denominator behind a counterA fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Current readings: Hits / requests: 0 / 16; Hit rate: 0%; Address-stream period: 16 addresses. All parameters are adjustable with the labeled controls below.Fully associative LRU cache after 16 deterministic requests: 0 hits and 16 misses.Most recent 16 requestsaddress = i × 3 mod 160M3M6M9M12M15M2M5M8M11M14M1M4M7M10M13MCurrent cache order · oldest → newest471013Latest address 13 · miss: inserted as newestToy address stream · period 16 · starts empty
Cached addressMiss / evictionMost recent request
Hits / requests
0 / 16
Hit rate
0%
Address-stream period
16 addresses
t = 0 s

A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA count and its denominator must describe the same scope before a ratio or comparison has meaning.

Check your understanding

An inner plan node displays actual rows=5 and loops=1,000. Ignoring rounding, which row-count interpretation is correct?

Try it yourself

A scan estimates 400 emitted rows and observes 100 per execution. Calculate positive-count q-error. What should you report if it instead observes zero?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use the larger of actual/estimate and estimate/actual only when both are positive.

Compare with the solution

For actual 100 and estimate 400, q=max(100/400,400/100)=4. If actual is zero, that expression is undefined; report the zero-versus-400 discrepancy and an absolute error of 400 rows, or disclose an explicit smoothing convention. Silently substituting one can conceal a qualitatively different empty-result case.

Take it with you

  • Compare estimated and actual rows within the same invocation scope.
  • Node timings are inclusive and buffer events are not direct device-latency measurements.
  • Locate consequential early errors and control instrumentation, parameters, and workload conditions.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.