advanced29 min studyLesson 300 in Engineering

Evolve meaning as carefully as bytes

Separate physical carriers, logical types, schema reconciliation, and feature support when introducing new analytical file writers and readers.

On this page

What you’ll understand

  • Distinguish byte-level readability from preservation of logical value meaning.
  • Build a feature matrix covering types, encodings, codecs, pages, and reader versions.
  • Plan a rollout that tests old and new files and preserves recovery-compatible retained data.

A reader can parse a file successfully and still interpret its values incorrectly. An integer carrier might mean a signed count, a decimal's unscaled value, or a timestamp with a particular unit. Format evolution must therefore preserve both bytes and meaning across a population of files and consumers. Compatibility is a relationship among concrete writers, features, schemas, and readers, not one universal property conferred by a familiar file extension.

Physical storage does not determine logical meaning

Parquet separates a compact set of physical types from logical annotations that specify interpretation. An INT64 carrier alone does not tell an application whether a number is an epoch count in microseconds or a scaled decimal. Decimal precision and scale, timestamp units and instant/local semantics, signedness, and string ordering all matter. Changing only a label can silently change query values or pruning order without changing any payload bits. A migration should explicitly transform values where needed and test boundaries, rather than treating metadata edits as a generally safe conversion.

Nested schema changes add further questions about presence and identity. Adding an optional field can permit old records to be represented with missing values under an engine's reconciliation policy, but the file format alone does not assign every application's default or guarantee every reader combines mixed schemas automatically. Renames need a table or engine's field-identity rules; matching names by guesswork can conflate unrelated columns. Validate null versus absent/default semantics and preserve list structure. Physical type widening must also respect range, precision, and supported conversion behavior in each consumer.

vdecimal=u 10−s,nnanos=1000 nmicrosv_{\rm decimal}=u\,10^{-s},\qquad n_{\rm nanos}=1000\,n_{\rm micros}
u is an unscaled integer and s the decimal scale. The timestamp conversion applies to the same instant/epoch interpretation when multiplying microsecond counts into nanoseconds, with representable-range checks; a changed annotation alone does not perform this numerical conversion.↔ Scroll wide equations horizontally.

Support is a feature matrix, not one version switch

List the physical and logical types, encodings, codecs, data-page variants, statistics semantics, and optional indexes the writer can emit. Then test each actual consumer version against that set. A reader may support a codec but not a new logical type, or correctly read values while ignoring an optional pruning index. Ignoring an optional performance feature can be safe; inventing the meaning of an unsupported required feature is not. Parquet's documented fallback for an unrecognized annotation does not guarantee an application that needs its semantics can safely accept raw physical values.

File-format version, library release, table metadata version, and Arrow in-memory or IPC feature support are different layers. Do not infer that upgrading one enables all others, or that an engine advertising Parquet support implements every optional feature. Consult current implementation documentation and retain representative compatibility fixtures. Include mixed dictionary/plain pages, null-heavy nested records, maximum integer values, decimal scales, timestamp units, and missing statistics. Check decoded values and query answers rather than only whether an open call succeeds; incorrect statistics interpretation can corrupt results by pruning valid data.

Femitted⊆⋂r∈RFcorrectly supported(r)F_{\rm emitted}\subseteq\bigcap_{r\in R}F_{\rm correctly\ supported}(r)
For required reader population R, the writer's emitted features must lie within their correctly supported intersection for the promised operation. Support includes semantic interpretation, not merely header parsing; optional feature ignorance is acceptable only where its specified fallback preserves correctness.↔ Scroll wide equations horizontally.

Roll out readers and writers with retained-file evidence

Introduce compatible readers before enabling writer features that older required consumers cannot handle. Write a small canary generation, validate it with the full reader matrix, compare query answers with established data, and publish through the table's checked commit protocol. Keep a rollback strategy that accounts for newly created files: reverting a writer binary does not rewrite files it already emitted. Long-lived snapshots and backups can retain old schemas and codecs for years, so the supported read history and restoration process deserve explicit fixtures and ownership.

Use the cache instrument only to illustrate a bounded working-set effect during compatibility scans. Reset, set both C=8 and s=2, predict eight hits among the first sixteen requests, then Play or Step. Reset and re-enter both C=4 and s=2 for a separate run with no hits. This deterministic stream caps at 4,096 requests and has no files or type system; a hit is not evidence that a reader understands a logical type. Its limited lesson is to keep cache-warm timing separate from semantic pass/fail evidence when rerunning a compatibility fixture suite.

Two metadata changes that are not value conversions

An integer carrier stores 12,567 with decimal scale two. Another carrier stores 1,234 microseconds from an agreed epoch. Explain what must happen to preserve values when decimal scale changes to three and timestamp unit changes to nanoseconds.

  1. At scale two, 12,567 represents 125.67. Reinterpreting the same integer at scale three would produce 12.567, so preserving 125.67 requires a new unscaled integer 125,670, subject to precision and carrier-range checks.

  2. The same epoch offset of 1,234 microseconds is 1,234,000 nanoseconds. Retagging 1,234 as nanoseconds would change the offset by a factor of one thousand.

  3. Validate both conversions through required readers and the table schema policy, including negative values and extremes. Preserve instant/local interpretation and reject or explicitly handle values that no longer fit the target representation.

The result

Correct evolution changes numerical representation when units or scale require it, rather than assuming a metadata label preserves value meaning automatically.

Explore the model

Interactive lab

Interactive lab · cache

Warm compatibility scans still need semantic checks

Reset, set both C=8 and s=2, predict eight hits in the initial sixteen requests, then Play or Step. Reset and re-enter C=4 and s=2 to compare no hits over that eight-address cycle. The stream caps at 4,096 and models only toy LRU reuse; it cannot validate logical types, codecs, schemas, or format compatibility.

Warm compatibility scans still need semantic checksA fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Current readings: Hits / requests: 0 / 16; Hit rate: 0%; Address-stream period: 16 addresses. All parameters are adjustable with the labeled controls below.Fully associative LRU cache after 16 deterministic requests: 0 hits and 16 misses.Most recent 16 requestsaddress = i × 3 mod 160M3M6M9M12M15M2M5M8M11M14M1M4M7M10M13MCurrent cache order · oldest → newest471013Latest address 13 · miss: inserted as newestToy address stream · period 16 · starts empty
Cached addressMiss / evictionMost recent request
Hits / requests
0 / 16
Hit rate
0%
Address-stream period
16 addresses
t = 0 s

A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA faster repeated fixture read can reflect caching; compatibility evidence comes from exact interpreted values and required query behavior.

Check your understanding

A new reader can parse an INT64 payload but does not understand the logical type the application requires. What is the sound application-level conclusion?

Try it yourself

An unscaled integer 345 represents a decimal with scale two. A migration changes to scale four while preserving value. Find the required new integer and explain one additional compatibility check.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

First identify the represented value, then solve for the new unscaled integer.

Compare with the solution

The value is 3.45. At scale four it requires unscaled integer 34,500 because 34,500 × 10⁻⁴ = 3.45. Check that declared precision and the physical carrier can represent the new integer, and that every required reader interprets the new annotation and mixed historical files correctly. Retagging 345 alone would instead produce 0.0345.

Take it with you

  • Physical carriers require logical type, units, and schema rules to preserve meaning.
  • Compatibility must cover the actual emitted feature set and every required reader's semantics.
  • Reader-first rollouts, canary files, retained-history fixtures, and checked publication make evolution reviewable.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.