Run state machines: separate logical intent from a worker incarnation

Define states such as ready, deciding, waiting-for-approval, executing, reconciling, cancelling, and terminal outcomes. Specify which events permit each transition and which component owns them. Waiting can be a healthy durable condition rather than a process failure. Terminal success should require the declared effect and cleanup conditions, not merely the model emitting a final sentence. Record unresolved external outcomes separately so a timeout cannot silently become either success or a claim that nothing happened.

Give a logical user intent a stable run identity, then distinguish a worker incarnation or execution generation from a tool step and its retry attempts. A replacement worker should not accidentally create a new business effect merely because its process identifier changed. Temporal, for example, distinguishes Workflow IDs and Run IDs under its own lifecycle rules; that terminology is not identical to every proposed incarnation design. Choose identities by purpose and document which survive retries, restarts, or explicit new user intents.

A proposed state store can condition an update on the expected revision and current incarnation. Two workers reading the same revision must not both commit incompatible next states. After one succeeds, the other rereads or abandons its stale decision. The storage system must actually enforce that condition atomically. A timestamp comparison or a process-local mutex does not provide the same cross-worker property. Include event identity so duplicate delivery does not advance a transition twice under the chosen protocol.

An incarnation fence protects only components that check it. If an old worker has already sent an external write to a service that ignores the fence, rejecting its later completion event does not undo the write. External effects need their own idempotency, version, or reconciliation mechanism. Keep a late result as evidence of what occurred even if it cannot drive the new incarnation's state transition directly. A durable run can therefore enter reconciliation rather than pretend every stale message is harmless.

Enumerate transitions for duplicate completions, cancellation races, delayed approvals, worker replacement, and a tool result arriving after a terminal decision. Verify which events are retained, ignored for control flow, or sent to reconciliation. Separate a retryable step failure from a failed logical run. Measure recovery and unresolved-effect age as well as completion rate. A model can choose sensible actions while the runtime mishandles them, and a correct state machine can faithfully execute a poor plan; those need different evaluations.

The Markov lab illustrates state progression only. Reset, set p=1 and q=0, predict state-one probability one after one Step, then Step once. Separately Reset, set p=0.50 and q=0, predict probability 0.75 after two Steps, then Step twice. It starts in state zero and has no revisions, workers, or atomic transitions. Its exact probabilities cannot prove durability or fencing. Use it to distinguish an abstract transition model from the actual identity and storage protocol being tested.