Reclaiming tombstones: deletion needs stable knowledge
Trace resurrection from an old replica and derive a fixed-membership stable frontier without replacing proof with elapsed time.
On this page
The question for today
What you’ll understand
- Construct a resurrection history after premature deletion-metadata removal.
- Compute a componentwise stable frontier from acknowledgments.
- Explain why old replicas, backups, and membership epochs affect safe reclamation.
A tombstone records that a value or insertion was removed. Keeping it forever can be expensive, but deleting it too early can let an old copy restore content that should remain gone. Garbage collection is therefore a distributed knowledge problem: what evidence shows that no admissible future message can reintroduce the forgotten history? Elapsed wall-clock time alone cannot answer that question when replicas or backups may be disconnected for an unbounded interval.
Absence does not carry the same information as removal
In the simple retained-history OR-Set, an old add tag remains harmless because every receiver that knows its tombstone treats it as removed. If a replica drops the tombstone but still accepts the old add from another participant, the element can become visible again. Deleting the local add record as well does not solve the general problem: an offline peer or delayed message may retain it. Before compacting, account for every allowed source of stale history, including durable queues, restored snapshots, and clients that synchronize after a long absence.
A safe design can retain compact causal context that proves the old event was observed and removed, establish a stable delivery boundary under fixed membership, or fence old histories through an epoch and require returning replicas to resynchronize. These are different mechanisms. A compressed representation without explicit tombstones is not automatically one without historical knowledge. Likewise, dropping a replica from the current membership list is unsafe if its old messages remain admissible under the same identities and protocol rules.
Stable frontiers require an actual acknowledgment model
Under fixed membership, suppose each replica reports a prefix frontier of operations it has durably applied. Taking the componentwise minimum gives a frontier known to have been applied by every member, provided acknowledgments are truthful, monotone, and scoped to the same history. An event below that frontier has reached all those members. Message stability, meaning every member applied an event, is weaker than causal stability in some operation-based protocols, where no future delivered operation is concurrent with that event. Whether its tombstone can be removed additionally depends on the type’s protocol: delayed messages must be rejected or interpreted using retained context so that forgetting one record cannot reenable a removed event.
A minimum over only reachable members gives a different claim. An offline member omitted from the calculation may still have an old add and no corresponding removal. Dynamic membership therefore needs a defined reconfiguration and state-transfer protocol. A rejoining actor that lost its counters must not silently reuse its previous identity. Epoch fencing can prevent old state from participating, but the fence has to be checked on messages and restored storage, not merely documented in an operator runbook.
Operational grace periods encode assumptions
Database tombstone grace periods are practical policies that depend on repair, outage, and restoration assumptions. Cassandra documents the need to coordinate tombstone removal with repair to avoid deleted data returning. This is not a claim that every CRDT uses Cassandra’s mechanism. For your own protocol, specify the maximum supported offline interval, repair coverage, accepted snapshot epochs, and the rule for rejecting or rebuilding stale replicas. Monitor violations of those assumptions rather than treating the configured number of seconds as a correctness theorem.
Use the reliability lab to separate reachable majorities from universal knowledge. Reset, set replica count n=3 and unavailable count k=1, predict two up with a majority threshold of two, then Step. A majority is available, but the absent third replica may still hold old state. The lab counts availability only; it has no tombstones, acknowledgments, epochs, or reclamation algorithm. Changing k does not prove that everyone learned a deletion. The worked frontier calculation supplies the stronger evidence required under its stated membership model.
Work it through
An offline replica blocks a stable boundary
Members A, B, C report applied-prefix vectors (5,4), (5,3), and (2,3). A removal has actor event ID (first actor,4). Compute the stable frontier and test that event’s coverage.
Take minima by component: min(5,5,2)=2 and min(4,3,3)=3. The stable frontier is (2,3), not the componentwise maximum (5,4).
The removal’s first-actor counter four exceeds stable counter two, so the acknowledgments do not show that C has applied it. A and B knowing the removal cannot substitute for C’s missing prefix.
After C durably advances to (5,3), the minimum becomes (5,3), covering the removal event. This supplies delivery knowledge, but actual tombstone compaction still needs the type’s stale-message/context rule and a policy preventing unfenced old snapshots from returning.
The initial stable frontier does not cover the removal. A majority’s knowledge and a safe full-membership reclamation boundary are different facts.
The living notebook
Explore the model
Interactive lab · reliability
A reachable majority is not universal knowledge
Reset, set n=3 and k=1, predict two up and majority threshold two, then Step. This model counts availability only; it does not track deletions, acknowledgments, epochs, or safe garbage collection.
- Available replicas
- 4 of 5
- Majority needed
- 3
- Majority available?
- Yes
A replica count and a majority threshold, without a consensus protocol. Playback highlights fixed replica states. Having a majority alone does not establish linearizability, durability or failure independence. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forThe unavailable member can retain stale history even while a majority serves requests.
Pause & reason
Check your understanding
Your turn
Try it yourself
Three durable prefix acknowledgments are (6,2), (4,5), and (5,3). Compute their stable frontier. Does it cover first-actor event five?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Take minima componentwise across every member.
Compare with the solution
The stable frontier is (4,2). It does not cover first-actor event five, because one member has acknowledged only the first four events. Even after all members cover five, a reclamation rule must still account for delayed messages and restored state using the actual datatype protocol.
Keep the idea
Take it with you
- Removal metadata prevents stale additions from looking new again.
- A stable delivery frontier uses all relevant durable acknowledgments.
- Reclamation also needs correct stale-message, restoration, and membership rules.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.