advanced23 min studyLesson 274 in Engineering

Covering indexes: payload availability still needs visibility

Explain included columns, MVCC visibility checks, and the leaf-width tradeoff behind PostgreSQL index-only scans.

On this page

What you’ll understand

  • Distinguish search-key columns from included payload columns.
  • Explain why an index-only plan can still visit table tuples.
  • Calculate the space and visibility-work effects of a hypothetical covering index.

A covering index keeps the values a query needs close to its search keys, potentially avoiding separate table access. That benefit depends on more than selecting the right column list. The index access method must provide the required values, the database must establish visibility, and the extra leaf payload must justify its storage and maintenance cost.

Store a result without making it a search key

Suppose a query finds an account by account_id and returns its display_name. An index whose key is account_id and whose included payload is display_name can provide both values without making the name part of the ordered search boundary. Included values do not contribute the same navigation or ordering capabilities as key columns. In PostgreSQL 18, a unique index on account_id with an included name enforces uniqueness on the key, not on the account-and-name pair. This distinction matters when preserving an application invariant.

Covering is relative to a query, not an index's universal property. A query needing another nonstored value may still require table access. Expressions, partial predicates, collations, and access-method support can affect eligibility. An included value is duplicated data whose changes must be maintained correctly; it is not a pointer that magically fetches the latest table value at no cost. Read the actual plan and confirm that projected values, filters, and required ordering fit the chosen access path.

Lold=⌊P−HE⌋,Lcover=⌊P−HE+I⌋L_{\rm old}=\left\lfloor\frac{P-H}{E}\right\rfloor,\qquad L_{\rm cover}=\left\lfloor\frac{P-H}{E+I}\right\rfloor
P−H is usable leaf-page bytes, E is the original complete entry width, and I is the additional included-payload width in bytes under a fixed-width model. Both capacities count entries and ignore compression changes, variable lengths, and extra alignment unless already included in E and I.↔ Scroll wide equations horizontally.

MVCC visibility is separate from stored values

In PostgreSQL's MVCC model, finding an index entry is not sufficient to establish that the referenced row version is visible to the current snapshot. An index-only scan can consult a compact visibility map: if the corresponding heap page is marked all-visible, it can avoid a heap visibility check for that candidate. Otherwise it must visit the heap tuple. Thus the plan-node name does not guarantee zero heap fetches during execution. Recent updates can clear useful visibility information even when every requested value is present in the index.

Separate tuple visibility visits from distinct page reads. Several candidates can reside on one cached heap page, so a count of 100 heap fetches is not necessarily 100 physical device operations. Visibility-map access also has a cost, usually on a smaller structure, while cache state and concurrency affect timing. A relatively stable table can make covering attractive; a rapidly changing table may offer less heap avoidance. Observe actual execution metrics under representative snapshot lifetimes rather than estimating the benefit from index eligibility alone.

Hvisits=Ncandidates(1−v)H_{\rm visits}=N_{\rm candidates}(1-v)
v is the fraction of candidate visits whose heap pages are established all-visible in this simplified count model. H_visits counts candidate heap-visibility visits, not distinct pages or physical reads. If v is an estimated probability, the right-hand side is an expectation; visibility changes during execution require more detailed accounting.↔ Scroll wide equations horizontally.

Pay for the duplicate payload deliberately

Wider leaf entries increase page count, cache pressure, insertion work, and the chance that an entry exceeds the access method's size limit. PostgreSQL can omit included payload from upper routing levels, so the leaf penalty need not widen every internal separator. Included columns can still affect update behavior and implementation optimizations; for example, PostgreSQL 18 does not apply B-tree deduplication to INCLUDE indexes. Compare the actual workload's saved heap work with the extra maintenance and retained storage instead of adding every projected column by default.

The cache lab illustrates a larger address working set without representing included columns. Reset, set C=4 and s=4, predict twelve hits among the sixteen initial requests, then Play or Step. For a separate experiment, Reset, set C=4 and s=2, predict zero initial hits because eight distinct addresses cycle through four slots, then Play or Step. The stream change is explicit; the lab cannot widen an index entry or inspect MVCC. Use it to explain why a layout that saves table visits can still reduce cache reuse elsewhere.

Balance leaf width and heap avoidance

A fictional leaf has 4,000 usable bytes. Original entries are 40 bytes; including another value adds 40 bytes. A query produces 1,000 candidates, with exactly 90% of their visits covered by all-visible heap pages.

  1. Original leaf capacity is 4,000/40=100 entries. Covering capacity becomes 4,000/80=50, so a packed leaf layer needs twice as many pages for the same entry count.

  2. Of 1,000 candidate visits, 900 have all-visible evidence and 100 require heap visibility visits. These are candidate visits rather than a guaranteed 100 physical reads.

  3. Measure whether the saved table work outweighs the extra index footprint and writes. The arithmetic does not decide the plan's latency, especially if heap pages are cached or visibility changes frequently.

The result

The index offers a clear source of saved heap work and a clear leaf-space penalty. Both belong in the decision, while key-only uniqueness remains a separate semantic property.

Explore the model

Interactive lab

Interactive lab · cache

More page addresses can change reuse

Reset, set C=4 and s=4, predict 12 hits in the initial 16 requests, then Play or Step. Separately Reset, set C=4 and s=2, predict zero hits, then Play or Step. This deterministic LRU working-set comparison does not change index widths, execute SQL, or inspect visibility maps.

More page addresses can change reuseA fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Current readings: Hits / requests: 0 / 16; Hit rate: 0%; Address-stream period: 16 addresses. All parameters are adjustable with the labeled controls below.Fully associative LRU cache after 16 deterministic requests: 0 hits and 16 misses.Most recent 16 requestsaddress = i × 3 mod 160M3M6M9M12M15M2M5M8M11M14M1M4M7M10M13MCurrent cache order · oldest → newest471013Latest address 13 · miss: inserted as newestToy address stream · period 16 · starts empty
Cached addressMiss / evictionMost recent request
Hits / requests
0 / 16
Hit rate
0%
Address-stream period
16 addresses
t = 0 s

A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forSaving one access path can still enlarge another working set; covering needs a complete workload comparison.

Check your understanding

Why might PostgreSQL report heap fetches for an Index Only Scan?

Try it yourself

A query has 2,500 candidate visits and an estimated all-visible fraction of 0.8. Estimate heap visibility visits. If forty candidates share each relevant heap page, can you call that estimate a physical-read count?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Multiply by one minus the fraction, then distinguish visits, pages, and cache misses.

Compare with the solution

The estimate is 2,500×0.2=500 candidate heap visits. It is not a physical-read count: candidates can share pages, pages can already be cached, and the stated forty-per-page grouping would itself need to match the actual visited candidates. Visibility-map and index accesses remain outside this count.

Take it with you

  • Included values provide payload without equivalent search-key semantics.
  • Index-only eligibility does not remove MVCC visibility requirements.
  • Saved heap work must justify wider leaves and added maintenance.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.