Schema evolution and data contracts
Separate serialization compatibility from semantic correctness and migrate producers, readers, and historical data safely.
On this page
The question for today
What you’ll understand
- Distinguish backward and forward compatibility using explicit writer-reader pairs.
- Explain defaults, nullability, and semantic units during data evolution.
- Plan an expand-migrate-contract rollout including old retained records.
A producer adds a nullable field and a compatibility checker approves it. A model then interprets an old duration in seconds as milliseconds and silently changes its recommendations. The bytes remain readable while the meaning is wrong. A useful data contract therefore has two layers: how a writer encodes a record for a reader, and what the decoded values promise about identity, units, time, quality, and permitted use. Both layers need versioned evidence.
Name the writer and reader in every claim
Backward compatibility commonly means a new reader can consume data written with an older schema. Forward compatibility means an older reader can consume data from a newer writer. A full compatibility policy checks both directions; transitive policies extend checks beyond just the immediately preceding version. Always name the serialization format and registry policy, because a rule valid for Avro is not automatically valid for a different encoding or for handwritten JSON parsing.
Avro resolves writer and reader schemas under defined rules. A reader default can supply a field absent from older written records; it is not a universal instruction that every writer may omit that field. Nullable fields, unions, field aliases, and allowed type promotions each have format-specific behavior. Test representative old bytes with the actual new reader, not just two schema documents. Historical replay can activate versions that no currently running producer uses.
Version meaning as well as representation
A float named duration says little about whether it stores seconds, milliseconds, or a normalized score. Record units, valid ranges, null meaning, timestamp basis, and owner. Distinguish unknown from zero and not-applicable; replacing every null with zero can change a model feature's distribution. A newly added category value may be syntactically valid but break a downstream exhaustive business rule. Compatibility testing must therefore include the consumer behavior that depends on those values.
Treat semantic conversions as explicit transformations with provenance. If an old field changes meaning, introduce a distinct field or contract version rather than reusing its name silently. A data-quality check can reject negative physical durations, but a plausible value in the wrong unit may pass a range check. Compare distributions and paired converted examples. For machine learning, preserve the training-serving transformation identity so a schema rollout does not create a feature mismatch between historical data and live inference.
Expand, migrate, then remove with evidence
First expand readers to tolerate the new representation while still accepting the old one. Then deploy producers that populate the new field, optionally dual-writing during a bounded transition. Backfill historical records under a named transformation and reconcile counts and semantic samples. Finally remove old reads and writes only after consumer telemetry and retention policy establish that the old form is no longer required. A deployment rollback must still find a readable representation during the permitted rollback window.
Do not confuse consumer rollout completion with historical migration completion. Cold archived data, delayed jobs, and restored checkpoints can reintroduce old encodings. Keep a contract inventory that records producers, consumers, compatibility mode, retention horizon, and deprecation owner. Automate byte-level compatibility tests and semantic examples in the same release process. Avoid treating one successful compatibility check as permanent authorization for every future field change; each migration needs its own concrete mixed-version matrix.
Work it through
Migrate a duration feature without a silent scale change
A version-one record stores duration_s = 2.4. Version two introduces duration_ms. There are 5 million old records and a backfill validates 1,000 records/s. Ignore startup and retries.
The equivalent new value is 2.4 × 1,000 = 2,400 milliseconds. Copying 2.4 into the new field would change meaning by a factor of 1,000.
Deploy readers that prefer duration_ms but explicitly convert duration_s when the new field is absent, keeping the fallback measurable.
The backfill needs 5,000,000/1,000 = 5,000 seconds, or 83 minutes 20 seconds, under the constant validated-throughput assumption.
Reconcile converted samples and consumer fallback counts before contraction. Keep old readability for the declared replay and rollback horizons, even after the live backfill finishes.
A safe rollout combines mixed-version reader behavior, a measured historical migration, and explicit unit equivalence.
The living notebook
Explore the model
Interactive lab · vectors
Inspect components without confusing representation and meaning
The lab draws one 2D vector controlled by angle and magnitude and exposes its x/y projections. Use it as a reminder that one object has several related representations. It does not encode records, convert duration units, validate a schema, or prove compatibility between readers and writers.
- x component
- 1.393
- y component
- 0.975
- Magnitude
- 1.7
Playback traces a fixed vector and its projections. Angle and magnitude set the geometry; the moving marker does not change them. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forRepresentations can differ while referring to one quantity, but their relationship must be specified and checked.
Pause & reason
Check your understanding
Your turn
Try it yourself
An old record lacks a new optional risk_score field. A reader supplies default 0. What questions must be answered before treating that as a legitimate zero-risk observation for model evaluation?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Separate successful schema resolution from the interpretation of a missing measurement.
Compare with the solution
Determine whether zero denotes measured absence of risk, an unknown value, or merely a serialization fallback. Track missingness separately when it carries information, and verify that training and serving use the same policy. Evaluate old and new cohorts separately to detect rollout artifacts. A readable default does not create an observed label or measurement, and silently treating it as one can bias evaluation.
Keep the idea
Take it with you
- Compatibility is directional and must name concrete writer and reader versions.
- Decoded fields still need stable semantic units, null meaning, and quality rules.
- Contraction waits for live consumers, retained data, replay, and rollback requirements.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.