Deployment state machines
Model desired state, observed evidence, guarded transitions, and crash recovery so retries do not create duplicated or unsafe release effects.
On this page
The question for today
What you’ll understand
- Specify guarded release states and invariants.
- Distinguish a retried request from a proven single external effect.
- Trace reconciliation after a crash between an action and its status update.
A deployment controller repeatedly observes an environment and tries to make it match a requested release. It may crash after changing a route but before recording success, or receive an old observation after a newer decision. A state-machine design makes those cases explicit: it defines what is allowed to happen, what evidence permits progress, and how the controller resumes without treating uncertainty as success.
State labels need guards and durable identities
A useful release machine might distinguish staged, validating, exposing, promoted, paused, aborting, and recovered states. These are an example design, not names that every controller must use. Each transition has a guard, such as matching artifact identity, sufficient readiness, fresh analysis for the current revision, and an authorized exposure change. Bind evidence to a release revision or generation so that a successful check from yesterday cannot promote today's image. The invariant is a property of the actual environment, not merely a status field containing a reassuring word.
Desired state and observed state can differ while work is in progress. An API update can be accepted before routing has propagated or workloads have started. A controller should compare the observed revision with the requested one and reconcile missing work. Kubernetes resource versions support conditional updates against concurrent changes, but that does not make a separate load-balancer update part of the same transaction. A process can also outlive its lease or receive a late response; fencing or equivalent target-side version checks are needed where stale actors could overwrite a newer release.
A crash can leave an action outcome unknown
Suppose the controller submits a traffic change and crashes before persisting the acknowledgment. On restart, blindly submitting a new non-idempotent action can duplicate an external effect. Instead use a stable operation identity where the target supports deduplication, and query actual target state to resolve uncertainty. Setting desired traffic to an absolute revision is often easier to reconcile than repeatedly applying a relative increment. A retry returning success does not establish exactly-once behavior unless the effect and deduplication semantics support that claim.
Idempotence means repeating the same operation has the same intended effect as applying it once under the stated domain. It does not mean that repeated unrelated operations are safe or that partial actions need no cleanup. Persist a progress record with the release and operation identities, make status updates conditional on the expected revision, and treat timeout as unknown until observed. Compensation must also be resumable: a crash halfway through abort should not re-enable exposure simply because the happy-path machine restarted at its first step.
Prove safety separately from eventual progress
Safety asks whether a forbidden state can occur, such as serving an unvalidated revision. Liveness asks whether an eligible release can eventually progress when required dependencies recover. Retrying indefinitely might preserve safety while never finishing; bypassing a failed guard might finish while violating safety. Specify bounded waiting, inconclusive analysis, cancellation, and operator intervention paths. Argo phases and sync waves can order work, but successful ordering is not a substitute for application-specific checks, replay-safe hooks, and observation of the target revision.
The Markov lab models only a simplified retry-completion probability. Reset, set both p=0.5 and q=0, predict state-one probability 0.5 after one Step, then Step. Separately Reset, set both p=0.25 and q=0, predict 0.25 after one Step, then Step. State zero begins with all probability and state one is absorbing for q=0. This exact two-state distribution does not execute a deployment machine, validate guards, or prove idempotence. Explain why a high modeled chance of eventual completion says nothing about whether a forbidden intermediate state was reached.
Work it through
Reconcile a promotion whose acknowledgment was lost
Release r17 requires setting canary traffic to an absolute ten percent. Operation key r17-expose-10 is persisted. The controller sends it, the target applies ten percent, and the controller crashes before recording completion. A repeated request has a documented deduplication contract. For this numerical exercise, the toy router assigns exactly its declared fraction of the next 10,000 requests.
On restart, read the desired release and the same operation key. Do not create a new key or issue a relative additional ten-percentage-point increment, which would incorrectly request twenty percent.
Observe the target's current revision and traffic. If it still reports r17 at ten percent, the intended effect is already present; its stipulated share is 0.10×10,000=1,000 candidate requests. Record that observation conditionally for r17 rather than applying a second increment.
If the acknowledgment remains uncertain, retry the identical keyed absolute request under the stated contract. Two deliveries still request ten percent, or 1,000 of the toy router's 10,000 requests, rather than twenty percent and 2,000. A conflicting newer revision requires reconciliation rather than overwriting it.
The example depends on explicit target semantics and revision checks. A request counter or controller restart alone cannot prove the external effect happened only once.
The living notebook
Explore the model
Interactive lab · markov
Retry progress is not a safety proof
Reset, set both p=0.5 and q=0, predict state-one probability 0.5 after one Step, then Step. Separately Reset, set both p=0.25 and q=0, predict 0.25, then Step. This exact two-state retry analogy starts in state zero and cannot verify release guards, idempotence, or real controller transitions.
- P(state 1) at step 0
- 0%
- Stationary P(state 1)
- 60%
- Chain behavior
- Monotone convergence
Exact finite-step probabilities for a two-state chain starting in state 0, not sampled trajectories. Playback advances two transitions per second. When p = q = 0 every distribution is stationary; when p = q = 1 the chain alternates forever. A single zero transition creates an absorbing state. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forCompletion likelihood and the safety of every intermediate action are different claims.
Pause & reason
Check your understanding
Your turn
Try it yourself
For an independent retry model with success probability 0.25 per attempt and an absorbing completed state, calculate completion probability after three attempts. Why is that not a deployment safety guarantee?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The only incomplete path is failure on all three attempts.
Compare with the solution
The probability is 1−0.75^3=0.578125, or 57.8125 percent. It assumes independent equal-probability attempts and no regression after completion. A real release needs guarded state transitions, revision-bound evidence, idempotent effects, and compatibility; none of those properties follows from this two-state probability calculation.
Keep the idea
Take it with you
- Bind transition guards and observations to a specific release revision.
- Resolve unknown effects through stable identities and target-state reconciliation.
- Safety invariants and eventual progress require separate arguments.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.