Rollback compatibility and forward recovery
Choose recovery from the current durable state rather than assuming an older binary, restored snapshot, or inverse migration automatically recreates a safe system.
On this page
The question for today
What you’ll understand
- Distinguish code rollback, traffic reversal, and data recovery.
- Identify when old code is incompatible with state created by a new version.
- Calculate an explicit recovery timeline and replay-capacity requirement.
A rollback changes some selected part of a release. It does not reverse time. Requests may already have written data, published messages, or called an external system. Recovery therefore begins by identifying the current state and the properties that must be restored. An older artifact is useful only when it can operate safely on that state and its dependencies remain available.
Write down what the reversal actually changes
Traffic reversal sends future work toward another pool; code rollback replaces executable bytes; configuration rollback restores selected settings. Each can be useful without undoing completed effects. A database column dropped by a committed migration is not recreated with its original contents merely because the old container starts. Even when DDL was transactional before commit, a later deployment reversal is a different operation. Record which state transitions have become externally visible and which recovery paths still preserve the application's invariants.
A rollback candidate needs a compatibility contract covering schema, serialized values, model assets, messages, credentials, and external APIs. New code may write values that an old reader rejects or interprets differently. A flag or rollout controller cannot repair that mismatch by reporting the previous revision as active. Keep old readers compatible through an intentional window when rollback is part of the plan. After crossing a destructive or semantic boundary, designate forward recovery or a tested restoration-and-reconciliation path instead of relying on an obsolete binary as the only fallback.
Forward recovery preserves progress when reversal cannot
A forward fix can make current data readable, stop the harmful writer, or repair affected rows while retaining legitimate new work. A repair needs its own identity, bounded batches, idempotent update rules, validation, and reconciliation. Distinguish compensating an external action from deleting an internal record: a payment refund or message correction is a new business event with its own failure cases, not a literal erasure of history. Keep an audit trail connecting the faulty effect and its repair so repeated recovery attempts do not repeat compensation.
Restoring a backup or a point-in-time database can recover an earlier consistent state, but that state may omit valid work performed afterwards. PostgreSQL point-in-time recovery requires appropriate base backups and WAL history; a logical dump is not the same recovery mechanism. Decide what writes must be preserved, whether replay is possible and safe, and how external systems will be reconciled. A technically successful restore can still violate the application's data-loss objective or duplicate already completed outside effects. Verify the recovered business state, not just database startup.
Rehearse the state-dependent decision
For a release, document a decision point: before this state change, a compatible old revision can be restored; after it, use a specified forward repair or restoration procedure. Test that boundary with data written by the new version. Include the time needed to find and authorize the recovery path, obtain retained artifacts, and validate user-facing correctness. A rollback button that was exercised only before any new writes is weak evidence for an incident occurring hours later. A controller's fast rollback window is a deployment mechanism, not an assertion about business-data reversibility.
The queue instrument illustrates a recovery-work capacity condition. Reset, set both λ=4 and μ=3, predict backlog growth of one normalized item per second from four, then Play or Step. Separately Reset, set both λ=2 and μ=3, predict drainage from four at t=4, then Play or Step. These are independent fluid runs, not live remediation. The model has no stored rows, replay semantics, or compensation identities. Explain why a repair service cannot clear a growing real backlog while harmful new work continues arriving faster than completed repairs.
Work it through
An older binary does not repair new-format rows
A faulty release leaves 60,000 affected rows in a format the retained old binary cannot read. A forward repair deployment takes sixty seconds, then repairs a fixed backlog at 300 completed rows/s, followed by forty seconds of validation. Assume the harmful writer is already stopped.
Starting the old binary changes code but leaves all 60,000 incompatible rows. Under the stated read contract, that action alone cannot restore correct service.
The bounded repair phase needs 60,000/300=200 seconds. Add the sixty-second deployment and forty-second validation for a sequential modeled recovery duration of 300 seconds, or five minutes.
If deployment instead takes 120 seconds, the same model totals 120+200+40=360 seconds. Neither timeline includes detection, decision, retries, or outside effects; the recovery procedure must validate those separately rather than advertise the arithmetic as an incident guarantee.
Choose a recovery path compatible with current state and budget every required phase. Reversing executable identity and repairing durable data are different operations.
The living notebook
Explore the model
Interactive lab · queue
Recovery capacity must exceed new repair demand
Reset, set both λ=4 and μ=3, predict one-item-per-second growth from four, then Play or Step. Separately Reset, set both λ=2 and μ=3, predict drainage from four at t=4, then Play or Step. This fluid analogy performs no rollback, data replay, repair, or compensation; M/M/1 assumptions are separate.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forStopping the source of harmful work and providing enough repair capacity are distinct parts of a recovery plan.
Pause & reason
Check your understanding
Your turn
Try it yourself
A replay queue contains 100,000 events. New eligible events arrive steadily at 500/s and the recovery consumer completes 1,000/s. Estimate the ideal drain time, then state one semantic condition beyond rate stability.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use net completion capacity, not the full consumer rate.
Compare with the solution
Net drain is 1,000−500=500 events/s, so the ideal time is 100,000/500=200 seconds. Replay also needs correct ordering or commutativity, deduplication or idempotent effects, and a valid snapshot/offset boundary. Clearing the queue quickly is not proof that every external or database effect was applied correctly.
Keep the idea
Take it with you
- Code rollback and traffic reversal do not rewind durable or external effects.
- Select recovery according to current read/write and dependency compatibility.
- Forward repair and restoration both need explicit state validation and reconciliation.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.