advanced28 min studyLesson 250 in Engineering

Adversarial traces: check every partial execution

Build replayable partition, delay, crash, and restart histories with safety oracles and explicit limits on explored executions.

On this page

What you’ll understand

  • Trace an old leader and a new majority without confusing local belief with commitment.
  • Write safety oracles for applied history, votes, and persistent state.
  • Scope bounded exploration and statistical stress evidence accurately.

The most useful consensus test often stops before a reassuring final state. Pause an old leader after two copies, elect another leader elsewhere, crash a voter between a write and its response, or install a snapshot while delayed messages remain in flight. A correct protocol must preserve its invariants throughout these partial executions. This lesson turns the preceding mechanisms into a disciplined trace exercise and distinguishes a counterexample search from an unbounded proof of the complete system.

Write a state table before interpreting a timeout

Record each node's role, current term, durable vote, durable log, known commit index, applied index, and active configuration. Record messages as separate objects with sender, receiver, term, request context, and payload. Delivering, duplicating, delaying, or dropping a message changes a specific transition; a timeout merely introduces another eligible event. This explicit representation stops the test author from silently granting a vote, changing an entry, or removing a member to make a desired narrative work.

A partition can leave an old leader believing it still leads while a majority elects a higher-term leader. The minority's uncommitted suffix may be replaced when communication resumes. Its already committed prefix must not be contradicted, and it must not serve unsupported linearizable reads while isolated. Track client invocation and response events too: a lost response can leave the client uncertain even after a command is committed. A test that only compares final database contents can miss both a stale read and an earlier invalid acknowledgement.

applied⁡u(i)=x∧applied⁡v(i)=y⟹x=y\operatorname{applied}_u(i)=x\land\operatorname{applied}_v(i)=y\Longrightarrow x=y
For correct replicas applying the same consensus log, any common applied index i must identify the same command. Indices and commands are abstract values. This safety oracle complements, rather than replaces, vote, commitment, membership, and client-history checks.↔ Scroll wide equations horizontally.

Crash cuts expose dependencies hidden by a happy path

Insert crash points before a required storage operation, after issuing it, after its durability completion, before sending the dependent reply, and after that reply is delivered. Recovery must use only the state the storage model says survived. Replaying all issued writes as if they were durable erases exactly the failures the test should investigate. Include old messages after restart, duplicate append requests, stale vote responses, and snapshots with different boundaries. The oracle must reject an impossible recovered state rather than quietly normalize it.

For membership tests, preserve the actual configuration activation rules and quorum predicates. A test harness that always uses the newest desired membership can accidentally make an unsafe implementation look correct. Likewise, an authority check mocked as an unconditional success removes stale-leader read failures. Keep deterministic seeds and a replay log, then minimize failing traces while preserving their invariant violation. A shorter counterexample is usually easier to turn into a precise regression and a corrected safety argument.

Nbinary=2mN_{\rm binary}=2^m
For exactly m fixed, ordered binary test decisions, there are 2^m decision sequences before reduction. This is a finite harness count, not a count of all distributed executions: reordering, new messages, crashes, storage outcomes, and timing choices expand the state space.↔ Scroll wide equations horizontally.

State what exploration proves and what it omits

A bounded model checker can exhaust its stated state space and establish that its modeled oracle has no counterexample within those bounds. It does not automatically cover more nodes, longer logs, unmodeled disk behavior, or integration code. Fairness and eventual-timing assumptions are needed when asking liveness questions; a test that drops every message forever cannot reasonably require every client request to finish. Conversely, allowing a system to stop indefinitely is not evidence that its progress mechanism meets an operational target.

The probability lab illustrates a separate, deliberately assumed sampling claim. Reset, explicitly set success probability to 0.10 and trials to ten, interpret success as observing a failure, predict the zero-success bar, then Play or Step. For another comparison Reset and explicitly set probability 0.10 and trials twenty. The exact binomial model assumes independent trials with a known common observation probability. Real test schedules can be correlated and their failure probability unknown, so these bars neither enumerate executions nor certify a passing consensus implementation.

Heal a minority leader's conflicting suffix

Five voters share a committed prefix through index 40. In term six, leader A appends X at 41 only on A and B; it remains uncommitted and unapplied. A partition isolates {A,B}. The connected {C,D,E} elects D in term seven and durably commits Y at 41 on all three.

  1. The term-six X has two copies, fewer than the five-node majority three, so the trace does not establish its commitment. A's continued local leader belief cannot turn those two acknowledgements into a quorum or authorize an unsupported fresh read.

  2. D's term-seven entry Y reaches three voters in its current term, satisfying the stated fixed-membership commitment rule under the earlier election and persistence invariants. The common committed prefix through 40 remains unchanged.

  3. After healing, A observes the higher term and steps down. A valid AppendEntries continuation matching the prefix through 40 replaces A's conflicting uncommitted term-six entry at 41 with term-seven Y. A must not have applied X earlier, and the test must not relabel X as committed merely because it once existed locally.

The result

The legitimate replacement concerns an uncommitted suffix. Applying X before commitment, serving a stale isolated-leader read, or overwriting a committed prefix would each be a distinct failing trace.

Explore the model

Interactive lab

Interactive lab · probability

Assumed detection chance is not schedule coverage

For independent toy comparisons use Reset, explicitly set both p=0.10 and n=10, predict the zero-success bar, then Play or Step; Reset and explicitly set p=0.10 and n=20 for another run. Success means hypothetical bug observation. The binomial model does not execute a protocol, explore crashes, or measure a real test's failure probability.

Assumed detection chance is not schedule coverageThese are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Current readings: Mean · np: 5; Variance · np(1−p): 2.5; Total probability: 1. All parameters are adjustable with the labeled controls below.03581000.070.150.220.3Successes kP(X = k)k = 0 · P = 9.77e-4
Exact probabilityHighlighted outcomeMean np
Mean · np
5
Variance · np(1−p)
2.5
Total probability
1
t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forKeep a statistical sampling assumption distinct from exhaustive modeled coverage and from the protocol's safety proof.

Check your understanding

A bounded checker finds no applied-history violation for three nodes, two log positions, and a storage model that makes every write immediately durable. What conclusion is justified?

Try it yourself

A toy harness processes six fixed binary deliver-or-drop decisions in a predetermined order. How many decision sequences exist? Under a separate hypothetical independent trial model with failure-observation probability 0.1, what fraction of ten trials can show no failure?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Count binary decisions separately from probabilities. Neither calculation includes arbitrary message order or unknown real-world bug likelihood.

Compare with the solution

The harness has 2^6=64 binary decision sequences before reductions, with its stated processing order fixed. The independent sampling model gives 0.9^10≈0.3487, about 34.87%, for observing no failure. This probability does not follow from the 64-sequence count and does not estimate a real bug's likelihood. Larger histories, crash cuts, delayed storage completion, and reordered messages require explicit additional modeled actions and their own coverage argument.

Take it with you

  • Check invariant preservation during partial executions, not only after final convergence.
  • Model durable state and protocol activation rules faithfully at every crash cut.
  • Bounded exploration and statistical stress tests provide different, scoped evidence.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.