Coordinator failure and decision recovery

List failure points before prepare, after some participants prepare, after all yes votes, after the decision becomes durable, and after only some decision messages arrive. These states can look similar from one participant’s viewpoint. The coordinator’s recovery rule must distinguish them using durable information and the protocol’s documented defaults. A conventional coordinator may recover an undecided transaction by choosing abort when no commit decision was ever made, but an individual prepared participant cannot infer that absence merely because it cannot reach the coordinator.

Once commit is durably decided, recovery must preserve commit and redeliver it to required participants until they can complete. Aborting the remaining participants would not repair the failure; it would create disagreement with participants that already committed. Keep transaction identifiers stable across retrying decision delivery. A newly elected coordinator must be fenced from an obsolete owner and must read the authoritative state under the replication protocol. A fresh process with an empty memory map is not evidence that no decision exists.

A prepared transaction can retain locks and recovery metadata while waiting for resolution. PostgreSQL exposes prepared transaction identifiers and preparation times through pg_prepared_xacts. That inventory helps locate an external transaction manager’s unresolved work; it does not determine commit versus rollback by itself. Store enough application correlation to identify the owning workflow and decision authority. Avoid putting secrets or personally sensitive payloads into identifiers merely for convenient incident searches.

Resolution should be idempotent at the logical protocol level. A participant may process a commit decision and lose its acknowledgment, causing the coordinator to send the same decision again. Preserve enough terminal knowledge or reconciliation evidence to distinguish already completed work from a different transaction. Only forget coordinator state after the protocol’s acknowledgment and recovery conditions permit it. Prematurely deleting the record can turn an ordinary retry into an unresolvable ambiguous state, even if every server is otherwise healthy.

A replicated decision service may continue when one node fails if its consensus, storage, and fencing conditions hold. This improves the availability of evidence; it does not transform an undecided transaction into a committed one. Record both the decision-service epoch and transaction identity when the design requires them. Test coordinator restart after persistence but before reply, participant restart while prepared, and repeated decision delivery. The checker should inspect recovered decisions and effects, not merely whether the retry returned a success status.

In the reliability lab, press Reset, set replica count to 3 and unavailable count to 1, predict majority availability, then Step. For a separate run, Reset and explicitly set both counts to 3 and 2, predict, then Step. The model counts two versus one available replicas; it contains no consensus messages, persistent decisions, or fencing. Use that limitation to explain why a majority indicator is only one input to a decision-service availability argument, and cannot select a prepared transaction’s outcome.