advanced24 min studyLesson 279 in Engineering

Bloom filters: rule out membership under an intact contract

Derive the approximate false-positive model, choose a hash count, and preserve the conditions behind no false negatives.

On this page

What you’ll understand

  • Distinguish a negative membership result from a positive candidate.
  • Apply the standard approximate false-positive and optimal-hash formulas with their assumptions.
  • Explain why unsafe deletion, corruption, or a mismatched data version can create false negatives.

A Bloom filter spends a compact bit array to avoid some expensive exact lookups. In its ordinary insert-only form, it can say definitely absent or possibly present. The useful guarantee depends on complete insertion and consistent hashing into an intact filter for the correct data set. It is a conditional correctness property, not a promise that an arbitrary stored bit array can never lie.

Set bits conservatively, verify positive answers

Start with m zero bits and k hash positions per key. Inserting a key sets every selected bit to one. A query finding any required bit zero can reject membership because a correctly inserted key would have set it. If all are one, other keys might have set those same bits, so the answer remains a candidate requiring the exact underlying lookup. The filter does not store row values, prove uniqueness, or enumerate matching records. A positive answer can also be a genuine member rather than an error.

The no-false-negative argument assumes the inserted key is queried with the same hashing and encoding rules, every required insertion is reflected, and no relevant bit has been lost or improperly cleared. A filter for yesterday's file cannot safely reject keys added to a different current file unless the design accounts for them. Concurrent construction and publication must preserve that version agreement. Corruption, incomplete writes, incompatible serialization, and stale metadata can violate the premise; a negative result is useful only within the system's actual integrity and lifecycle contract.

pfp≈(1−e−kn/m)kp_{\rm fp}\approx\left(1-e^{-kn/m}\right)^k
m is filter bits, n is the number of distinct inserted keys in the model, and k is a positive integer hash-position count. This standard approximation assumes uniform hashing and approximate independence of queried bit states. It estimates a nonmember query's false-positive probability, not the fraction of all positive answers that are false.↔ Scroll wide equations horizontally.

Choose space and hash work together

After n insertions, more selected bits are set, so absent queries are harder to reject. Increasing m relative to n lowers saturation, while changing k has a tradeoff: more queried positions initially improve rejection but also set more bits during insertion and cost additional hash work. The familiar continuous optimum is proportional to bits per key times ln two. An implementation chooses a nearby feasible integer and may generate positions using a specified practical hashing scheme. The simple formula is not an exact finite-filter guarantee for every such scheme.

Expected avoided work depends on the query mix. If nearly all queries are real members, the filter adds checking overhead while rejecting little. If most queries are absent and exact lookup is costly, even a modest filter can help. A one-percent false-positive probability among nonmembers is not a one-percent error rate among all queries or positive results. Account for cache misses to the bit array, hashing CPU, and whether multiple underlying files require multiple filter checks. Choose the parameter budget for the workload rather than treating the lowest modeled probability as automatically optimal.

k⋆=mnln⁡2,E[F]=Qabsent pfpk^\star=\frac{m}{n}\ln2,\qquad \mathbb{E}[F]=Q_{\rm absent}\,p_{\rm fp}
k-star is the continuous optimum of the standard approximate model for positive m and n; actual k is an integer. Q_absent counts absent-key queries. The expected false-positive count uses their appropriate probability; independence across queries is not required for this expectation, though a binomial distribution needs stronger assumptions.↔ Scroll wide equations horizontally.

Deletion and rebuilding preserve a versioned set

Clearing the ordinary Bloom bits for a deleted key is unsafe because another live key may depend on those bits. A counting variant can support certain deletion protocols, but must prevent counter underflow, overflow, and deletion of unsupported keys. Rebuilding from an authoritative snapshot can remove obsolete influence if writes arriving during construction are handled and the new filter is published with the correct data version. An immutable storage file naturally supports a file-specific filter; moving or merging its records requires corresponding filter maintenance rather than reusing arbitrary old metadata.

The probability lab models hypothetical false-positive events, not the filter algorithm. Reset, set p=0.01 and n=20, predict a mean of 0.2 events, then Play or Step to inspect exact binomial bars. Separately Reset, set p=0.10 and n=20, predict mean two, then Play or Step. This assumes twenty independent trials with a common stipulated probability. It cannot set m or k, insert keys, or prove no false negatives. Relate the two event rates to avoided exact lookups, then state which filter-integrity premises the bars do not check.

Estimate a ten-bit-per-key filter

Use 10,000 filter bits for 1,000 distinct keys under the standard uniform-hashing approximation. Choose k=7 and consider 10,000 absent-key queries.

  1. The space ratio is ten bits per key, and the continuous optimum is 10 ln2≈6.93, making seven a nearby integer choice. The raw bit payload is 10,000/8=1,250 bytes.

  2. Compute the approximate false-positive probability as (1−exp(−7×1,000/10,000))^7≈0.00819, or about 0.819%. It is a nonmember-query probability under the model.

  3. Across 10,000 absent queries, the expected false-positive count is approximately 81.9. Each such positive still performs exact verification; correctly inserted real members must also be handled by the underlying lookup.

The result

The model predicts roughly eighty-two unnecessary exact lookups among ten thousand absent queries, not eighty-two incorrect returned records.

Explore the model

Interactive lab

Interactive lab · probability

Count hypothetical false-positive events

Reset, set p=0.01 and n=20, predict mean 0.2 events, then Play or Step. Separately Reset, set p=0.10 and n=20, predict mean 2, then Play or Step. This exact binomial model assumes independent common-probability events; it neither constructs a Bloom filter nor exposes bit-count or hash-count controls.

Count hypothetical false-positive eventsThese are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Current readings: Mean · np: 5; Variance · np(1−p): 2.5; Total probability: 1. All parameters are adjustable with the labeled controls below.03581000.070.150.220.3Successes kP(X = k)k = 0 · P = 9.77e-4
Exact probabilityHighlighted outcomeMean np
Mean · np
5
Variance · np(1−p)
2.5
Total probability
1
t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA false positive adds verification work; integrity and correct versioning are separate prerequisites for safe negative answers.

Check your understanding

Why is clearing every ordinary Bloom bit used by a deleted key unsafe?

Try it yourself

A filter has a modeled nonmember false-positive probability of 0.02. Among 50,000 queries, 40,000 are absent and 10,000 are genuine members. Estimate false positives and explain why 2% of all positive answers is not the right interpretation.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Apply the probability only to absent queries, then include genuine members among positive results.

Compare with the solution

Expected false positives are 40,000×0.02=800. Under intact insertion and lookup assumptions, all 10,000 genuine members also test positive, so the expected positive-count ratio suggests about 800/10,800≈7.41% false among positives, not 2%. That ratio of expectations is a workload-level approximation, not an exact expectation of a random ratio.

Take it with you

  • A negative answer rules out membership only under intact insertion, hashing, and version agreement.
  • False-positive probability is conditioned on an absent key.
  • Deletion and rebuilding need explicit invariants rather than arbitrary bit clearing.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.