Detecting corrupt bytes is different from repairing them

A client-supplied checksum can help verify upload integrity when the API validates it over the specified bytes with the stated algorithm and checksum type. Multipart checksums can have full-object or composite meanings according to the service contract. An ETag is not a universal substitute. Decide whether the checked representation is the intended object payload, an encoded part, or stored backend data, and retain enough algorithm and version information to compare like with like. A matching string with an unspecified scope is weak evidence.

A noncryptographic checksum targets accidental corruption patterns rather than an adversary who can replace both data and checksum. Even a cryptographic digest needs a trusted reference if malicious replacement is in scope. A mismatch can originate in data, metadata, the transfer path, or an incorrect version association, so inspect the evidence before choosing a repair source. Copying the first available replica because another failed its checksum can propagate stale data unless the replication protocol also establishes the intended generation.

Checking data only when clients read it leaves cold objects unexamined. Scrubbing schedules integrity work independently of application demand. In the Ceph Squid documentation, shallow scrubbing compares metadata while deep scrubbing also reads object data and checks integrity. The exact schedule and repair behavior depend on configuration and implementation. A configured weekly interval is not proof that every object completed a check during the last week if overload, scheduling, or unavailable devices prevented progress.

Measure completed coverage, age since the last successful check, unreadable fragments, and the time from detection to verified repair. Scans compete with foreground and recovery I/O, so increasing their frequency can affect client latency and repair capacity. Conversely, postponing them indefinitely allows latent errors to overlap with later failures. A useful operational budget includes enough scan throughput to meet the intended coverage objective under realistic competing work rather than assuming unused bandwidth is always available.

After detecting a bad fragment, establish a valid authoritative source or enough valid matching erasure-code fragments. Repair writes replacement data and should verify the result and restored placement. A clean checksum on an old generation is not a license to overwrite a committed newer one. Also investigate repeated repairs on a device: serving a read by using another replica can hide a failing component from a simplistic success-rate graph. Ceph's read-repair and scrub-related health signals illustrate why error counts and client success answer different questions.

Use the cache lab to inspect request coverage rather than corruption itself. Reset, set both C=4 and s=4, predict that the initial sixteen requests visit only identifiers 0,4,8,12, then inspect and Play or Step. Reset and set both C=4,s=1 for a separate stream that visits all sixteen identifiers initially. The deterministic cache contains no checksum, disk, or scrub, and playback is capped at 4,096 evaluated requests. Its lesson is that repeated hot accesses do not cover the unvisited population; actual scrub completion must provide that missing evidence.