advanced25 min studyLesson 176 in Engineering

Checksums detect disagreement; redundancy supplies a repair source

Separate detection, authentication, localization, and repair while reasoning about checksum coverage, scrubbing, and correlated failure.

On this page

What you’ll understand

  • State which bytes a checksum covers and what a mismatch establishes.
  • Distinguish accidental-error detection from authenticity and recoverability.
  • Estimate metadata overhead and explain why scrubbing and independent copies serve different roles.

Storage can return bytes without reporting an I/O error and still return the wrong contents. A checksum gives the reader another value against which to check them. That is a powerful detector, but detection is only one part of a protection story. We must also ask whether the expected checksum is trustworthy, whether the right generation and location were read, and where a verified replacement could come from if the check fails.

Define coverage and trust before choosing a digest

A checksum is computed over a defined sequence of bytes and compared with an expected value when those bytes are read or inspected. Btrfs normally checksums data and metadata, with data checksums stored separately from their data blocks. Other filesystems may checksum selected metadata without providing the same coverage for ordinary file payload. Always inspect the feature and file settings involved. A statement that a filesystem uses checksums is incomplete until you know which objects, paths, and modes are covered by the protection being claimed.

A mismatch establishes disagreement, not the complete cause. The payload may be damaged, the stored checksum may be damaged, or an addressing or version error may have paired the wrong objects. A successful comparison is also not proof against an attacker who can replace both data and an unprotected expected checksum. Authentication requires an appropriate trusted digest, signature, or message-authentication scheme and a trust chain. For model releases, binding the content digest to a trusted manifest and generation can address questions that a block-level accidental-error check was never designed to answer.

c=H(x),verify⁡(x′,c)=[H(x′)=c]c=H(x),\qquad \operatorname{verify}(x',c)=[H(x')=c]
H is a specified deterministic checksum function, x is the original byte sequence, and x' is the retrieved sequence. The Boolean equality tests agreement; it does not prove where damage occurred or that an adversary could not replace both values.↔ Scroll wide equations horizontally.

Detection needs an available, verified repair candidate

If only one damaged copy exists, a checksum can tell you that recovery is needed without containing enough information to reconstruct the missing contents. Redundant data, parity under its stated assumptions, or an independent backup can supply repair candidates. The system must verify the candidate rather than blindly copy from the first responding device. Redundancy that shares a failure source may reproduce the same corruption everywhere. Distinguish multiple names, multiple physical copies, multiple devices, and independently administered backups; they provide different protection against different events.

Scrubbing proactively reads allocated content to find latent problems before an ordinary application happens to touch it. Btrfs scrub can verify checksummed data and metadata and, when a valid redundant copy is available under the selected profile, repair from that copy. It is not a general filesystem structural checker or a substitute for backups. Scheduling matters: scrub consumes bandwidth and must finish often enough for the intended detection window. Monitor detected, corrected, and uncorrectable errors separately, and keep evidence that a repair candidate actually passed its check.

fdigest=hB,Prandom collision≈2−bf_{\rm digest}=\frac{h}{B},\qquad P_{\rm random\ collision}\approx 2^{-b}
h checksum bytes per B data bytes gives a raw payload overhead fraction before tree metadata. The collision estimate applies to a b-bit ideal uniformly distributed digest for a random independent erroneous value; real CRC error patterns and adversarial inputs need their actual algorithm-specific analysis.↔ Scroll wide equations horizontally.

Measure exposure without inventing independence

Suppose a block is damaged shortly after a scrub. It can remain latent until a later scrub or application read detects it. A second failure during that interval can eliminate the good copy the repair plan assumed. Shorter detection intervals reduce this particular exposure but cost I/O; they do not remove software bugs, common power events, or malicious deletion. Build a threat and fault model first, then assign each mechanism a role. Restore tests are needed because possessing a backup and reconstructing a usable application state are different accomplishments.

The binomial lab models independent trials with failure probability p and trial count n. Before changing n, predict that the expected affected count scales as np; before increasing p, predict probability mass shifting toward more failures. This is useful for understanding why scanning more independently exposed blocks finds more events in a simplified model. It cannot estimate a drive's real failure probability, reproduce checksum collisions, or model correlated firmware faults. Explain which shared causes would invalidate independence before applying the arithmetic to redundant copies.

Account for checksum payload and a repair decision

An illustrative format stores a four-byte checksum for every 4,096-byte data block. A 1 GiB file has no holes or compression. One block fails verification; one redundant copy passes. Ignore tree overhead.

  1. The file contains 1,073,741,824 / 4,096 = 262,144 data blocks. Four checksum bytes each require 1,048,576 bytes, exactly 1 MiB.

  2. Raw checksum overhead is 4/4,096 = 1/1,024, approximately 0.0977 percent of payload. Additional index and allocation metadata are excluded.

  3. For the mismatching block, use the verified redundant copy as a repair candidate under the system's recovery rules. If no copy passes, report unrecoverable disagreement and recover from a suitable backup instead of calling the checksum a reconstruction code.

The result

A small checksum overhead can make corruption observable, but successful repair still depends on valid redundancy and correctly identifying the expected contents.

Explore the model

Interactive lab

Interactive lab · probability

Independent exposure is an assumption

Adjust probability p and trial count n in an exact binomial distribution. Treat a success as one independently affected block. Predict the mean np and how mass shifts when p increases. This is not a filesystem failure-rate estimator, checksum algorithm, or model of correlated replicas.

Independent exposure is an assumptionThese are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Current readings: Mean · np: 5; Variance · np(1−p): 2.5; Total probability: 1. All parameters are adjustable with the labeled controls below.03581000.070.150.220.3Successes kP(X = k)k = 0 · P = 9.77e-4
Exact probabilityHighlighted outcomeMean np
Mean · np
5
Variance · np(1−p)
2.5
Total probability
1
t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forDetection arithmetic becomes misleading when shared failures or untrusted expected checksums violate its assumptions.

Check your understanding

A block checksum mismatch is detected on a filesystem with only one data copy and no usable backup. Which conclusion is justified?

Try it yourself

A format uses a 32-byte digest per 8,192-byte data block. Estimate raw digest overhead as a percentage and explain one reason this percentage does not quantify data-loss probability.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Divide digest bytes by data bytes, then separate space cost from the fault model.

Compare with the solution

The fraction is 32/8,192 = 1/256, or 0.390625 percent before metadata overhead. That space ratio does not describe failure frequency, correlated faults, digest trust, or availability of a valid repair copy. Those require a separate model and operational evidence.

Take it with you

  • Checksum protection is meaningful only when its byte coverage and expected-value trust are explicit.
  • A mismatch detects disagreement; repair requires a verified source of correct contents.
  • Scrubbing reduces latent-error exposure but cannot replace independent backups and restoration tests.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.