Checksums detect disagreement; redundancy supplies a repair source
Separate detection, authentication, localization, and repair while reasoning about checksum coverage, scrubbing, and correlated failure.
On this page
The question for today
What you’ll understand
- State which bytes a checksum covers and what a mismatch establishes.
- Distinguish accidental-error detection from authenticity and recoverability.
- Estimate metadata overhead and explain why scrubbing and independent copies serve different roles.
Storage can return bytes without reporting an I/O error and still return the wrong contents. A checksum gives the reader another value against which to check them. That is a powerful detector, but detection is only one part of a protection story. We must also ask whether the expected checksum is trustworthy, whether the right generation and location were read, and where a verified replacement could come from if the check fails.
Define coverage and trust before choosing a digest
A checksum is computed over a defined sequence of bytes and compared with an expected value when those bytes are read or inspected. Btrfs normally checksums data and metadata, with data checksums stored separately from their data blocks. Other filesystems may checksum selected metadata without providing the same coverage for ordinary file payload. Always inspect the feature and file settings involved. A statement that a filesystem uses checksums is incomplete until you know which objects, paths, and modes are covered by the protection being claimed.
A mismatch establishes disagreement, not the complete cause. The payload may be damaged, the stored checksum may be damaged, or an addressing or version error may have paired the wrong objects. A successful comparison is also not proof against an attacker who can replace both data and an unprotected expected checksum. Authentication requires an appropriate trusted digest, signature, or message-authentication scheme and a trust chain. For model releases, binding the content digest to a trusted manifest and generation can address questions that a block-level accidental-error check was never designed to answer.
Detection needs an available, verified repair candidate
If only one damaged copy exists, a checksum can tell you that recovery is needed without containing enough information to reconstruct the missing contents. Redundant data, parity under its stated assumptions, or an independent backup can supply repair candidates. The system must verify the candidate rather than blindly copy from the first responding device. Redundancy that shares a failure source may reproduce the same corruption everywhere. Distinguish multiple names, multiple physical copies, multiple devices, and independently administered backups; they provide different protection against different events.
Scrubbing proactively reads allocated content to find latent problems before an ordinary application happens to touch it. Btrfs scrub can verify checksummed data and metadata and, when a valid redundant copy is available under the selected profile, repair from that copy. It is not a general filesystem structural checker or a substitute for backups. Scheduling matters: scrub consumes bandwidth and must finish often enough for the intended detection window. Monitor detected, corrected, and uncorrectable errors separately, and keep evidence that a repair candidate actually passed its check.
Measure exposure without inventing independence
Suppose a block is damaged shortly after a scrub. It can remain latent until a later scrub or application read detects it. A second failure during that interval can eliminate the good copy the repair plan assumed. Shorter detection intervals reduce this particular exposure but cost I/O; they do not remove software bugs, common power events, or malicious deletion. Build a threat and fault model first, then assign each mechanism a role. Restore tests are needed because possessing a backup and reconstructing a usable application state are different accomplishments.
The binomial lab models independent trials with failure probability p and trial count n. Before changing n, predict that the expected affected count scales as np; before increasing p, predict probability mass shifting toward more failures. This is useful for understanding why scanning more independently exposed blocks finds more events in a simplified model. It cannot estimate a drive's real failure probability, reproduce checksum collisions, or model correlated firmware faults. Explain which shared causes would invalidate independence before applying the arithmetic to redundant copies.
Work it through
Account for checksum payload and a repair decision
An illustrative format stores a four-byte checksum for every 4,096-byte data block. A 1 GiB file has no holes or compression. One block fails verification; one redundant copy passes. Ignore tree overhead.
The file contains 1,073,741,824 / 4,096 = 262,144 data blocks. Four checksum bytes each require 1,048,576 bytes, exactly 1 MiB.
Raw checksum overhead is 4/4,096 = 1/1,024, approximately 0.0977 percent of payload. Additional index and allocation metadata are excluded.
For the mismatching block, use the verified redundant copy as a repair candidate under the system's recovery rules. If no copy passes, report unrecoverable disagreement and recover from a suitable backup instead of calling the checksum a reconstruction code.
A small checksum overhead can make corruption observable, but successful repair still depends on valid redundancy and correctly identifying the expected contents.
The living notebook
Explore the model
Interactive lab · probability
Independent exposure is an assumption
Adjust probability p and trial count n in an exact binomial distribution. Treat a success as one independently affected block. Predict the mean np and how mass shifts when p increases. This is not a filesystem failure-rate estimator, checksum algorithm, or model of correlated replicas.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forDetection arithmetic becomes misleading when shared failures or untrusted expected checksums violate its assumptions.
Pause & reason
Check your understanding
Your turn
Try it yourself
A format uses a 32-byte digest per 8,192-byte data block. Estimate raw digest overhead as a percentage and explain one reason this percentage does not quantify data-loss probability.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Divide digest bytes by data bytes, then separate space cost from the fault model.
Compare with the solution
The fraction is 32/8,192 = 1/256, or 0.390625 percent before metadata overhead. That space ratio does not describe failure frequency, correlated faults, digest trust, or availability of a valid repair copy. Those require a separate model and operational evidence.
Keep the idea
Take it with you
- Checksum protection is meaningful only when its byte coverage and expected-value trust are explicit.
- A mismatch detects disagreement; repair requires a verified source of correct contents.
- Scrubbing reduces latent-error exposure but cannot replace independent backups and restoration tests.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.