Write-ahead logging and recovery
Trace the ordering that makes acknowledged writes recoverable and calculate a realistic replay and restoration budget.
On this page
The question for today
What you’ll understand
- Explain log-before-page ordering and the meaning of a durable commit.
- Separate crash replay, replication, and recovery from a backup.
- Calculate replay demand while accounting for continued log generation.
A database updates a buffer in memory, acknowledges a transaction, and loses power before writing that data page. Whether the acknowledged change survives depends on a carefully enforced ordering between its log and its pages. Write-ahead logging makes recovery possible without synchronously flushing every changed page at every commit. It does not excuse an unreliable storage stack, a missing backup, or an application that interprets an uncertain network response as definite failure.
Protect the order, not just the bytes
The essential rule is that the recovery information describing a page change must reach durable storage before the changed page can safely reach durable storage. A synchronous durable commit also requires the relevant commit log records to be flushed according to the configured policy before acknowledgement. The database can later write dirty pages in a different order because recovery can replay the ordered log. Group commit amortizes one flush across multiple transactions without changing their durability contract.
Durability is a promise involving software, the operating system, the device, and its power-loss behavior. A completed write into a volatile device cache is not necessarily a stable write. Configuration that delays commit flushing may deliberately permit loss of recently acknowledged transactions after a crash. Distinguish that tradeoff from disabling protective synchronization altogether. PostgreSQL documents these settings separately; inspect the exact deployed policy rather than assuming all successful SQL commits offer identical failure guarantees.
Understand what a checkpoint changes
A checkpoint establishes a recovery starting relationship between persisted database state and the log. It reduces the log work needed after a crash, but checkpoint completion does not mean that no later log is required. Recovery identifies the appropriate starting point and replays necessary records, using page and transaction metadata to avoid treating an uncommitted change as visible. Exact algorithms differ by engine; PostgreSQL's WAL should not be described as implementing every detail of another database's recovery algorithm.
Crash recovery on surviving storage differs from point-in-time recovery after lost storage or operator error. The latter needs a suitable base backup and an uninterrupted retained sequence of required WAL, plus the target recovery policy. Replication supplies another copy of changes, which may also include the accidental deletion you wish to undo. Test restoration into an isolated environment and verify application-level invariants, rather than declaring a backup useful merely because an object exists.
Budget catch-up and retained history
A recovering replica must consume old log while the primary continues creating new log. If replay capacity does not exceed generation rate, lag cannot shrink under the constant-rate model. Measure both rates in the same physical units: logical changed-row bytes are not automatically WAL bytes. Full-page images, index changes, and transaction metadata affect generated volume. Some records replay more slowly than others, so bytes per second is a workload-dependent planning measurement.
Retain enough history for the actual outage and recovery interval, and monitor retention pressure separately from replay lag. A stalled replication slot can preserve WAL until disk fills; an aggressive retention cap can instead make a consumer require reinitialization. Checkpoint frequency trades background write pressure and WAL behavior against restart work. Use measured restoration and replay drills to choose it, including the time to provision storage, fetch backup objects, and validate the recovered service.
Work it through
Recover a replica while writes continue
A replica trails by 90 GB of WAL. Replay sustains 300 MB/s and the primary generates 120 MB/s. Use decimal units. After replay catches up, validation takes six minutes while replay continues keeping the replica current.
Convert 90 GB to 90,000 MB so units match the measured rates.
Net lag reduction is 300 − 120 = 180 MB/s; dividing by total replay rate would overstate recovery speed.
Catch-up requires 90,000/180 = 500 seconds, or 8 minutes 20 seconds.
Add 360 seconds of validation while replay stays current: 860 seconds, or 14 minutes 20 seconds. The 90 GB backlog was measured at replay start; provisioning delays before that measurement are excluded.
The rate model predicts 14 minutes 20 seconds from replay start to a validated replica, provided retention remains available and replay keeps pace during validation.
The living notebook
Explore the model
Interactive lab · queue
Separate log generation from replay
Interpret equal-sized items as log work. Vary arrival λ and service μ in max(0,4+(λ−μ)t). The separate M/M/1 residence estimate assumes Poisson arrivals and exponential single-server service. This fluid demonstration has no WAL records, checkpoints, crash semantics, or disk persistence.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forCatch-up consumes spare replay capacity; persistence guarantees require an ordering protocol beyond queue draining.
Pause & reason
Check your understanding
Your turn
Try it yourself
A consumer is 24 GB behind and replays 100 MB/s while writes generate 80 MB/s. Find ideal catch-up time, then explain the effect of generation rising to 110 MB/s.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use replay minus generation, with decimal GB converted to MB. Check whether the denominator remains positive.
Compare with the solution
At 80 MB/s generation the spare rate is 20 MB/s, so catch-up is 24,000/20 = 1,200 seconds, or 20 minutes. At 110 MB/s generation, lag grows by 10 MB/s. There is no finite catch-up time at those constant rates; reduce generated work or increase effective replay capacity before relying on the original estimate.
Keep the idea
Take it with you
- Durable commit depends on enforced log ordering and the configured flush policy.
- Crash restart, replica catch-up, and point-in-time restoration solve different failures.
- Recovery estimates must include continued log production and validated restoration.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.