Journaling protects a filesystem invariant, not every transaction
Reason about journal commit, replay, and checkpointing while separating structural recovery from an application's multi-file consistency contract.
On this page
The question for today
What you’ll understand
- Explain why logging before installation can make metadata updates recoverable.
- Distinguish journal commit from checkpointing and application acknowledgment.
- Construct a multi-file invariant that metadata journaling alone cannot guarantee.
Updating a file can require several related metadata changes: allocate blocks, adjust a mapping, and change file length. A crash between arbitrary writes could leave those changes inconsistent. A journal provides a recovery protocol for a selected set of updates. The important question is what the protocol includes. A healthy, mountable filesystem after recovery does not necessarily contain a coherent application checkpoint or every byte a process recently wrote.
Record an operation before installing its pieces
Imagine a transaction describing several metadata updates. A write-ahead journal first records enough information to recover that transaction, then establishes a valid commit record under its ordering rules. Only a committed transaction is eligible for replay. Its updates can later be installed at their ordinary home locations, a process called checkpointing. If a crash interrupts that later installation, recovery can reapply the committed description rather than guessing which subset of independent writes represents the intended result. The distinction between a recoverable log and a final home location is central.
The journal itself needs protection against incomplete writes and ordering violations. A commit marker cannot make unavailable earlier contents magically reconstructible; the protocol must ensure required records and relevant data have reached its persistence boundary first. Checksums and record validation can reject incomplete or damaged transactions during replay. Replaying is also designed to be safe under the implementation's rules, including the possibility that some home writes already happened. Treat these as precise filesystem mechanisms rather than assuming any file named journal automatically makes updates atomic.
Choose the invariant whose recovery matters
Ext4's ordinary metadata-journaling modes do not turn arbitrary application data writes into one atomic transaction. Ordered mode imposes relationships between associated data writeout and metadata commitment, while writeback mode gives weaker data ordering; full data journaling includes file data in the journal. These choices affect cost and recovery behavior, but none should be translated casually into a portable promise that every unsynchronized overwrite survives. Identify the actual mount configuration, documented synchronization semantics, and failure model when evaluating an application protocol.
Suppose weights.bin contains a model and manifest.json contains its version and checksum. Filesystem metadata can be structurally consistent while the manifest names a newer generation than the data that survived. The filesystem has preserved its own invariants but the application pair is inconsistent. One solution is immutable generation files followed by a carefully synchronized publication step; another application may use a durable transaction log and recovery logic. The essential requirement is to state the allowed post-crash states before choosing operations, rather than hoping background journal timing supplies the missing transaction boundary.
Journal space creates a second flow to monitor
Logging can add writes, and the journal has finite space. If checkpointing cannot retire committed transactions fast enough, a workload may stall waiting for reusable log capacity even when data admission initially looked fast. Batching can amortize a commit's synchronization cost across several operations, at the expense of how long each waits before acknowledgment. Measure transaction size, commit latency, checkpoint progress, and application acknowledgment separately. A throughput improvement from grouping commits does not authorize acknowledging a group before its required durable boundary completes.
Use the queue lab as a log-pressure illustration: arrival and service count equal-sized transaction records admitted and safely retired per time unit. Press Reset, set both arrival to four and service to two, predict increasing backlog from four records, then Play or Step. For a separate retirement comparison, Reset again, set both arrival to one and service to three, predict drainage from four, then Play or Step. Editing a rate retains elapsed time and recalculates the formula from the initial four records; it does not continue the previous backlog under new rates. The lab has no replay engine, journal transaction boundaries, or filesystem data mode.
Work it through
Group commit changes cost without changing acknowledgment order
Assume each group needs a 4 ms durable log synchronization and contains twenty application operations. Ignore all other costs and assume operations are ready together. Compare with synchronizing each operation separately.
Twenty separate synchronizations take 20 × 4 ms = 80 ms in this serialized model, for 250 operations per second.
One group synchronization takes 4 ms and covers twenty operations, giving an idealized 5,000 operations per second and 0.2 ms of synchronization cost amortized per operation.
All twenty operations still wait for the group's 4 ms synchronization before a durable acknowledgment. The 0.2 ms amortized cost is not an individual 0.2 ms durability latency, and collecting a group can add waiting.
Batching reduces synchronization work per operation; it does not eliminate the commit dependency or turn metadata recovery into a transaction across arbitrary application files.
The living notebook
Explore the model
Interactive lab · queue
Committed records still need safe retirement
Reset, set both λ=4 and μ=2, predict growth from four equal-sized records, then Play or Step. For a separate comparison, Reset, re-enter both λ=1 and μ=3, predict drainage from four, then Play or Step. Each Reset starts afresh; slider edits retain elapsed time. This log-pressure analogy cannot demonstrate replay correctness or data-mode guarantees.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forCommit, installation, reclamation, and application acknowledgment have different roles even when their rates interact.
Pause & reason
Check your understanding
Your turn
Try it yourself
A journal admits 600 equal-sized records each second and retires 450 each second. If 3,000 record slots are initially free, estimate time until those slots are consumed under a constant-rate fluid model.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use the difference between admission and safe retirement, not admission alone.
Compare with the solution
Backlog grows by 600 − 450 = 150 records per second. Three thousand free slots are consumed in 3,000/150 = 20 seconds if rates remain fixed and no earlier throttling occurs. Real journals use transaction boundaries and implementation-specific reservation rules, so this calculation motivates measurements rather than predicting an exact stall time.
Keep the idea
Take it with you
- Write-ahead ordering makes selected updates recoverable through committed log records.
- Filesystem structural consistency is weaker than an application's multi-object consistency invariant.
- Group commit amortizes work but preserves the required durable acknowledgment boundary.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.