advanced24 min studyLesson 276 in Engineering

Bitmap access: combine candidate sets, then recheck what was lost

Distinguish persistent bitmap structures from transient bitmap scans, trace AND/OR combinations, and account for lossy page-level candidates.

On this page

What you’ll understand

  • Compute intersections and unions of candidate row-location sets.
  • Explain why a bitmap heap scan loses index ordering.
  • Distinguish exact tuple locations, lossy page candidates, predicate rechecks, and MVCC visibility.

A bitmap lets an engine combine candidate locations from multiple searches before visiting table data. That can turn scattered accesses into a page-oriented visit order. It does not mean every database stores a permanent bitmap index, and it does not mean every bit proves a visible matching row. Identify what a bit represents and what information the plan must recheck.

Separate the index type from the scan strategy

A persistent bitmap index is a stored representation whose design may associate bit vectors with values or ranges, often with compression. A PostgreSQL Bitmap Index Scan instead builds a temporary in-memory candidate-location structure from an ordinary supported index access path. Several such results can be combined by BitmapAnd or BitmapOr, then consumed by a Bitmap Heap Scan. Calling that plan a persistent bitmap index conflates the search structure with the execution strategy. Their write behavior and storage requirements can be very different.

For an exact row-location model, an AND keeps only locations reported by every contributing predicate; an OR keeps locations reported by at least one. Duplicate locations in an OR are visited once in the set representation. Candidate row identities must refer to compatible table storage and snapshot processing, not arbitrary identifiers from unrelated tables. A combined candidate still needs the normal visibility and any required predicate checks. Set algebra organizes where to look; it does not create transaction semantics or make an approximate access method exact.

BAND=BA∧BB,BOR=BA∨BB,∣A∪B∣=∣A∣+∣B∣−∣A∩B∣B_{\rm AND}=B_A\land B_B,\qquad B_{\rm OR}=B_A\lor B_B,\qquad |A\cup B|=|A|+|B|-|A\cap B|
A and B are finite sets of candidate tuple locations in one table, and their exact bit vectors share the same location mapping. Counts are dimensionless. Lossy page entries require a broader candidate interpretation and later checks.↔ Scroll wide equations horizontally.

Lossy location information requires rechecking

Under memory pressure, a transient bitmap can represent a whole heap page rather than every matching tuple offset on that page. This lossy entry means that relevant candidates may be somewhere on the page, not that every row satisfies the original condition. The executor must inspect the page's tuples and recheck the necessary predicates. Access methods can also return lossy candidates for other reasons. Do not infer that a visible Recheck Cond means every page was lossy; examine exact and lossy page statistics and the access method's contract.

MVCC visibility is a separate check from predicate truth. A row location can pass the indexed condition but reference a version the current query cannot see. An exact tuple bitmap does not remove visibility processing, and a lossy bitmap does not authorize returning false matches. Conversely, lossy representation must preserve a superset of required candidates so matching rows are not omitted. When evaluating memory settings, compare added recheck work and page visits against memory consumption rather than treating a smaller bitmap as automatically faster.

T⊆C,R={r∈C:visible⁡(r)∧predicate⁡(r)}T\subseteq C,\qquad R=\{r\in C:\operatorname{visible}(r)\land\operatorname{predicate}(r)\}
C is a conservative candidate set and T contains all visible true matches required by the query. Rechecking computes result R under the snapshot. Correctness requires no required match be excluded from C; the formula does not model duplicate SQL bag semantics or join multiplicities.↔ Scroll wide equations horizontally.

Page order trades away index order

Visiting heap pages in physical order can improve locality, but it discards the original B-tree result order. A query requiring ORDER BY may therefore need a separate sort, and a small LIMIT may favor an ordered index scan that stops early. Combining two selective indexes also requires performing both searches and building the bitmap. A planner can rationally choose one index plus filtering, a composite index, or a sequential scan instead. The useful choice depends on candidate density, page distribution, cache state, and sorting costs.

The cache lab isolates page reuse rather than building a bitmap. Reset, set C=4 and s=4, predict twelve hits in the initial sixteen requests, then Play or Step. Separately Reset, set C=4 and s=1, predict zero hits because the sixteen-address cycle exceeds capacity, then Play or Step. The model cannot reorder a real candidate list, expose lossy tuple bits, or perform rechecks. Use it to explain why the mapping from rows to physical pages matters, while doing the actual candidate intersections on paper.

Combine exact candidate locations

One index yields candidate row locations A={1,3,4,8}; another yields B={3,4,6}. Assume these are exact locations in one table and handle visibility separately.

  1. For predicate A AND B, intersect the sets to obtain {3,4}, containing two candidate locations.

  2. For predicate A OR B, take {1,3,4,6,8}, containing five locations. The count also follows 4+3−2=5, so overlapping locations are not double-counted.

  3. If a lossy representation expands one candidate to its whole page, examine additional tuples and recheck predicates before returning visible matches. Neither the original set order nor SQL ORDER BY is preserved automatically.

The result

The exact AND has two candidates and the OR has five. These counts concern locations before any remaining visibility or predicate processing.

Explore the model

Interactive lab

Interactive lab · cache

Location density and reuse are separate from set algebra

Reset, set C=4 and s=4, predict 12 initial hits out of 16 requests, then Play or Step. Separately Reset, set C=4 and s=1, predict zero hits, then Play or Step. This LRU address stream does not combine bitmaps, reorder heap visits, or implement predicate rechecks.

Location density and reuse are separate from set algebraA fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Current readings: Hits / requests: 0 / 16; Hit rate: 0%; Address-stream period: 16 addresses. All parameters are adjustable with the labeled controls below.Fully associative LRU cache after 16 deterministic requests: 0 hits and 16 misses.Most recent 16 requestsaddress = i × 3 mod 160M3M6M9M12M15M2M5M8M11M14M1M4M7M10M13MCurrent cache order · oldest → newest471013Latest address 13 · miss: inserted as newestToy address stream · period 16 · starts empty
Cached addressMiss / evictionMost recent request
Hits / requests
0 / 16
Hit rate
0%
Address-stream period
16 addresses
t = 0 s

A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forCandidate-set correctness and the locality of resulting page visits require different reasoning.

Check your understanding

What does a lossy heap-page bitmap entry normally require?

Try it yourself

Two exact candidate sets have 120 and 80 locations, with 30 in common. Find AND and OR sizes. If a lossy candidate page has 40 visible rows but only three satisfy the condition, how many of those rows may be returned?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Apply inclusion-exclusion, then distinguish visibility from predicate truth.

Compare with the solution

The AND contains 30 locations, and the OR contains 120+80−30=170. On the lossy page, only the three visible rows satisfying the condition may be returned for that predicate. Forty visible rows do not imply forty matches; the other thirty-seven must be rejected by rechecking.

Take it with you

  • Transient bitmap scans are distinct from stored bitmap index formats.
  • Lossiness broadens candidates and requires rechecking without omitting true matches.
  • Page-oriented visits improve some access patterns while losing index ordering.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.