Data, storage, and the meaning of a committed write
Connect schemas, constraints, transactions, indexes, retention, and recovery to the promises an application makes.
On this page
The question for today
What you’ll understand
- Distinguish logical records from physical storage costs.
- Explain the purpose and limits of a transaction.
- Describe why replicas and backups solve different problems.
Data outlives the process that first creates it. Storage design therefore asks what must remain true after concurrent requests, crashes, schema changes, and human mistakes. Start from the meaning of a record and the invariants connecting records. Then choose the physical organization and recovery procedures needed to uphold those promises under stated failure conditions.
Model the invariant, then the representation
A schema gives records a structure, but domain rules give that structure meaning. A job may need a unique identifier, a valid owner, and a state transition that cannot skip required validation. Database constraints can enforce some rules close to stored data, where multiple application clients cannot accidentally bypass a shared assumption. Application checks still matter for usability and rules that cannot be expressed as a simple constraint.
Raw record size is only a starting point. Row metadata, indexes, write-ahead logs, free space, retained versions, replicas, and backups add storage. An index can reduce lookup work while increasing write work and capacity requirements. Decide which queries matter before adding indexes indiscriminately. For AI datasets, retain provenance and schema versions so a training record's origin and interpretation remain understandable later.
Transactions group changes under explicit guarantees
A transaction allows a group of changes to succeed together or be abandoned together. Moving a job from pending to running while recording its owner is easier to reason about when readers cannot observe only half of the intended change. Database systems provide particular isolation levels, so concurrent behavior still needs examination. Merely surrounding code with transaction syntax does not prove that every multi-request invariant is protected.
Durability also depends on the database's documented settings and failure assumptions. An acknowledged write may have different survival properties under different synchronization and replication policies. A transaction in one database does not automatically include an external message or object-store upload. When a workflow crosses those boundaries, define reconciliation or a coordinated pattern rather than claiming a single local commit made the entire distributed workflow atomic.
Recovery must be demonstrated
A replica may help availability when one machine fails, but it can also reproduce an accidental deletion. A backup preserves recoverable history according to its policy. Recovery point objectives describe acceptable data loss in time; recovery time objectives describe acceptable restoration delay. Both need a concrete failure scenario and a tested procedure. A backup file that has never been restored supplies weak evidence that recovery will work.
Practice restoration in an isolated environment, validate record counts and critical invariants, and measure how long the process takes. The lab counts replicas and unavailable replicas, showing whether the remaining count reaches a simple majority. Increase unavailable replicas at a fixed total and locate the threshold. This count alone says nothing about data freshness, quorum protocols, independence, database isolation, or backup recovery. Ask which protocol and failure assumptions a real system would need before a majority could support a consistency or durability claim.
Work it through
Size a retained event stream
A service stores 200 events per second, averaging 500 payload bytes each, for seven days. It maintains three equal full payload copies in this capacity model.
Convert retention to seconds: seven times 24 times 3,600 gives 604,800 seconds. The incoming raw payload rate is 200 × 500 = 100,000 bytes per second.
Multiply rate by retention: 60,480,000,000 bytes, or 60.48 GB of raw retained payload. Three equal copies give 181.44 GB before other storage costs.
Add separate budgets for indexes, logs, metadata, growth, and backups. Define a recovery test because three copies alone do not establish protection against correlated loss or a propagated deletion.
The explicit payload estimate is 60.48 GB, or 181.44 GB across three copies. Recovery promises require more than capacity arithmetic.
The living notebook
Explore the model
Interactive lab · reliability
Count replicas before claiming a guarantee
Choose replica count n and unavailable count k. Compare n−k with majority floor(n/2)+1. This is only a count; it does not simulate replication, prove consistency, or establish that stored data can be restored.
- Available replicas
- 4 of 5
- Majority needed
- 3
- Majority available?
- Yes
A replica count and a majority threshold, without a consensus protocol. Playback highlights fixed replica states. Having a majority alone does not establish linearizability, durability or failure independence. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forHaving a majority online is a prerequisite in some protocols, not a complete durability or recovery guarantee.
Pause & reason
Check your understanding
Your turn
Try it yourself
Records arrive at 50 per second with a 2,000-byte average payload. Calculate one day's raw payload, then identify two excluded capacity costs.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
One day is 86,400 seconds; keep bytes and records distinct.
Compare with the solution
The payload rate is 100,000 bytes per second, yielding 8,640,000,000 bytes, or 8.64 GB per day. Indexes and transaction logs are two excluded costs; replicas, retained versions, and backups may add more. Compression could reduce physical size, but must be measured on representative data.
Keep the idea
Take it with you
- Store domain invariants, provenance, and versions alongside useful data.
- Transaction guarantees have a scope and an isolation model.
- Replicas improve some failure cases; tested backups enable historical recovery.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.