advanced26 min studyLesson 263 in Engineering

Detect deadlocks across shard boundaries

Build a global wait-for argument while separating real cycles, stale observations, timeouts, and transaction generations.

On this page

What you’ll understand

  • Construct a wait-for cycle spanning multiple lock managers.
  • State when a cycle is sufficient to imply resource deadlock.
  • Avoid treating stale graph edges or timeouts as proof of deadlock.

Two shards can each report a perfectly ordinary local wait while the combined application is stuck forever. One transaction owns a lock on shard A and waits on shard B; another owns the needed lock on B and waits on A. Detecting this requires a model of dependencies across ownership boundaries, together with enough identity and temporal information to avoid diagnosing an obsolete collection of observations as a current deadlock.

Build the correct dependency graph

Use one vertex for each live transaction instance and a directed edge from a waiting transaction to the transaction whose incompatible lock prevents it from proceeding. For an AND-style nonpreemptive resource model, where each blocked operation needs its requested resource and locks are not spontaneously released, a directed wait-for cycle identifies a deadlock. Other waiting models need different reasoning: waiting for any one of several alternatives is not equivalent to requiring them all. State the model before applying a cycle rule.

A local shard sees only the dependencies involving resources it manages. The global cycle can be invisible in every local subgraph. Include transaction generation and resource identity in exported edges, especially when transaction identifiers or connections can be reused. Distinguish a transaction waiting for another lock owner from a transaction waiting for coordinator recovery or network delivery. Combining all these waits in one unlabeled graph can produce a plausible picture with no valid deadlock theorem behind it.

T1→T2→⋯→Tm→T1T_1\to T_2\to\cdots\to T_m\to T_1
Each arrow denotes a current incompatible-lock wait under the stated AND-style, nonpreemptive model. A cycle has m directed edges; it is not merely a cycle in an arbitrary service-dependency diagram.↔ Scroll wide equations horizontally.

Observe a changing graph carefully

Edges appear and disappear while a detector gathers information. A union of old observations can contain a cycle that never existed at one instant. A detector therefore needs a protocol and correctness assumptions, such as validated current edges, transaction epochs, or a formally specified distributed detection algorithm. A slow request or a lock-wait timeout can justify a bounded-wait policy without proving deadlock. Record whether an abort was caused by policy, a validated deadlock, or a different failure so operators can interpret the evidence.

When a real cycle is found, choose a victim using the system’s policy and ensure its abort actually releases the relevant resources before expecting peers to advance. An already prepared participant may be constrained by atomic-commit promises, so a lock manager cannot freely override the transaction protocol. Victim choice can balance wasted work and starvation, but it is not a new commit decision. Retrying should rerun the logical transaction under a bounded policy, with consistent lock ordering where practical to prevent repeated cycles.

Crestart=∑i∈AwiC_{\mathrm{restart}}=\sum_{i\in A}w_i
If victim set A is aborted, w_i is the completed work discarded for transaction i, measured here in CPU-milliseconds. This simple sum excludes lock-holding impact, I/O, and retry overhead; minimizing it alone can repeatedly starve long transactions.↔ Scroll wide equations horizontally.

Use cycles as evidence with explicit limits

Test two- and three-shard cycles with deterministic barriers: arrange the initial locks, confirm the waits, then invoke the detector. Also test a near-cycle whose first edge disappears before the last is created. Check that the detector does not confuse a retried transaction with its previous generation. Recovery tests should verify both the chosen victim’s final abort and eventual progress by survivors when the environment permits it. A graph screenshot without those state transitions is incomplete evidence.

The modular lab supplies a small cycle analogy. Reset, set modulus to 3 and additive step to 1, predict a return to zero after three additions, then press Step three times. For a new run, Reset and set both modulus to 4 and additive step to 2, predict the period, then Step twice. This is an exact arithmetic orbit, not a wait-for graph or deadlock detector. Explain why seeing a cycle here says nothing about whether any transaction is blocked or whether the observations were simultaneous.

Find a global cycle and its victim cost

T1 holds A and waits for B owned by T2. T2 waits for C owned by T3. T3 waits for A owned by T1. All waits are current and follow the nonpreemptive AND model. Their discarded-work costs are 9, 4, and 7 CPU-milliseconds. Assume any one can safely abort before prepare.

  1. Draw T1 → T2, T2 → T3, and T3 → T1. Each local lock manager may see only part of the three-edge global cycle.

  2. Aborting any one vertex breaks this simple cycle under the stated assumptions. Choosing T2 discards four CPU-milliseconds, less than nine or seven in this narrow cost metric.

  3. Confirm T2’s abort releases B before claiming T1 can proceed. Its later retry needs a policy that avoids repeatedly victimizing the same transaction.

The result

The current global cycle proves deadlock for this model. The cheapest stated victim is T2, but actual policies must also consider starvation and transaction state.

Explore the model

Interactive lab

Interactive lab · modular

A cycle is not automatically a deadlock

Reset, set modulus m = 3 and additive step a = 1, predict the return to zero, then Step three times. Compare a new Reset with both m = 4 and a = 2, predicting before two Steps. This additive orbit has no waits or lock owners.

A cycle is not automatically a deadlockAddition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Current readings: Residue r0: 0; Orbit period · m/gcd(m,a): 12; Distinct residues visited: 1 of 12. All parameters are adjustable with the labeled controls below.Repeated addition modulo 12: the current residue is 0, and the orbit period is 12.01234567891011+ 5mod 12An additive orbitn = 0rₙ = 0Period = 12Cycle starts at 0:0 → 5 → 10 → 3 →8 → 1 → 6 → 11 →4 → 9 → 2 → 7 → 0Exact residues · two additions per second
Additive orbitCurrent residue / next edge
Residue r0
0
Orbit period · m/gcd(m,a)
12
Distinct residues visited
1 of 12
t = 0 s

Addition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forCycles have meaning only after edges and state assumptions are defined; arithmetic recurrence does not diagnose resource deadlock.

Check your understanding

A detector unions wait-for edges collected at different times and sees a cycle. One edge disappeared before another was created. Which conclusion is justified?

Try it yourself

T1 waits for T2, T2 waits for T3, and T3 is running without waiting. Does this graph prove deadlock under the stated model? If T3 later waits for T1 while all earlier edges remain current, how many edges form the new cycle?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Follow dependencies until you reach a running transaction or return to a visited vertex.

Compare with the solution

The initial chain does not prove deadlock because T3 can make progress and potentially release the needed resource. Adding the current edge T3 → T1 creates a three-edge cycle. Under the nonpreemptive AND-lock assumptions, none can independently obtain its requested lock. A detector must verify the edges belong to these same live transaction instances and that no release already invalidated the chain.

Take it with you

  • Global deadlocks can span locally acyclic shard views.
  • Cycle detection requires a stated waiting model and temporally valid identities.
  • Victim abort, resource release, and retry progress are separate observable steps.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.