Detect deadlocks across shard boundaries
Build a global wait-for argument while separating real cycles, stale observations, timeouts, and transaction generations.
On this page
The question for today
What you’ll understand
- Construct a wait-for cycle spanning multiple lock managers.
- State when a cycle is sufficient to imply resource deadlock.
- Avoid treating stale graph edges or timeouts as proof of deadlock.
Two shards can each report a perfectly ordinary local wait while the combined application is stuck forever. One transaction owns a lock on shard A and waits on shard B; another owns the needed lock on B and waits on A. Detecting this requires a model of dependencies across ownership boundaries, together with enough identity and temporal information to avoid diagnosing an obsolete collection of observations as a current deadlock.
Build the correct dependency graph
Use one vertex for each live transaction instance and a directed edge from a waiting transaction to the transaction whose incompatible lock prevents it from proceeding. For an AND-style nonpreemptive resource model, where each blocked operation needs its requested resource and locks are not spontaneously released, a directed wait-for cycle identifies a deadlock. Other waiting models need different reasoning: waiting for any one of several alternatives is not equivalent to requiring them all. State the model before applying a cycle rule.
A local shard sees only the dependencies involving resources it manages. The global cycle can be invisible in every local subgraph. Include transaction generation and resource identity in exported edges, especially when transaction identifiers or connections can be reused. Distinguish a transaction waiting for another lock owner from a transaction waiting for coordinator recovery or network delivery. Combining all these waits in one unlabeled graph can produce a plausible picture with no valid deadlock theorem behind it.
Observe a changing graph carefully
Edges appear and disappear while a detector gathers information. A union of old observations can contain a cycle that never existed at one instant. A detector therefore needs a protocol and correctness assumptions, such as validated current edges, transaction epochs, or a formally specified distributed detection algorithm. A slow request or a lock-wait timeout can justify a bounded-wait policy without proving deadlock. Record whether an abort was caused by policy, a validated deadlock, or a different failure so operators can interpret the evidence.
When a real cycle is found, choose a victim using the system’s policy and ensure its abort actually releases the relevant resources before expecting peers to advance. An already prepared participant may be constrained by atomic-commit promises, so a lock manager cannot freely override the transaction protocol. Victim choice can balance wasted work and starvation, but it is not a new commit decision. Retrying should rerun the logical transaction under a bounded policy, with consistent lock ordering where practical to prevent repeated cycles.
Use cycles as evidence with explicit limits
Test two- and three-shard cycles with deterministic barriers: arrange the initial locks, confirm the waits, then invoke the detector. Also test a near-cycle whose first edge disappears before the last is created. Check that the detector does not confuse a retried transaction with its previous generation. Recovery tests should verify both the chosen victim’s final abort and eventual progress by survivors when the environment permits it. A graph screenshot without those state transitions is incomplete evidence.
The modular lab supplies a small cycle analogy. Reset, set modulus to 3 and additive step to 1, predict a return to zero after three additions, then press Step three times. For a new run, Reset and set both modulus to 4 and additive step to 2, predict the period, then Step twice. This is an exact arithmetic orbit, not a wait-for graph or deadlock detector. Explain why seeing a cycle here says nothing about whether any transaction is blocked or whether the observations were simultaneous.
Work it through
Find a global cycle and its victim cost
T1 holds A and waits for B owned by T2. T2 waits for C owned by T3. T3 waits for A owned by T1. All waits are current and follow the nonpreemptive AND model. Their discarded-work costs are 9, 4, and 7 CPU-milliseconds. Assume any one can safely abort before prepare.
Draw T1 → T2, T2 → T3, and T3 → T1. Each local lock manager may see only part of the three-edge global cycle.
Aborting any one vertex breaks this simple cycle under the stated assumptions. Choosing T2 discards four CPU-milliseconds, less than nine or seven in this narrow cost metric.
Confirm T2’s abort releases B before claiming T1 can proceed. Its later retry needs a policy that avoids repeatedly victimizing the same transaction.
The current global cycle proves deadlock for this model. The cheapest stated victim is T2, but actual policies must also consider starvation and transaction state.
The living notebook
Explore the model
Interactive lab · modular
A cycle is not automatically a deadlock
Reset, set modulus m = 3 and additive step a = 1, predict the return to zero, then Step three times. Compare a new Reset with both m = 4 and a = 2, predicting before two Steps. This additive orbit has no waits or lock owners.
- Residue r0
- 0
- Orbit period · m/gcd(m,a)
- 12
- Distinct residues visited
- 1 of 12
Addition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forCycles have meaning only after edges and state assumptions are defined; arithmetic recurrence does not diagnose resource deadlock.
Pause & reason
Check your understanding
Your turn
Try it yourself
T1 waits for T2, T2 waits for T3, and T3 is running without waiting. Does this graph prove deadlock under the stated model? If T3 later waits for T1 while all earlier edges remain current, how many edges form the new cycle?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Follow dependencies until you reach a running transaction or return to a visited vertex.
Compare with the solution
The initial chain does not prove deadlock because T3 can make progress and potentially release the needed resource. Adding the current edge T3 → T1 creates a three-edge cycle. Under the nonpreemptive AND-lock assumptions, none can independently obtain its requested lock. A detector must verify the edges belong to these same live transaction instances and that no release already invalidated the chain.
Keep the idea
Take it with you
- Global deadlocks can span locally acyclic shard views.
- Cycle detection requires a stated waiting model and temporally valid identities.
- Victim abort, resource release, and retry progress are separate observable steps.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.