Check schema compatibility in the actual reading direction

The writer schema describes how a record was encoded; the reader schema describes what the consuming application expects. In Avro, resolution relates those two schemas according to explicit rules. For example, a writer int can be promoted to a reader long, but a writer long is not generally compatible with a reader int merely because one sample happens to fit. Field names, unions, enum symbols, and defaults have their own rules. Pin the format version and implementation behavior used by your pipeline rather than assuming every serialization format gives the same meaning to adding or removing a field.

Backward compatibility commonly means that a new reader accepts older written data, while forward compatibility commonly means an older reader accepts new data. State those directions explicitly because terminology and transitive policies differ between tools. A canary that exercises only the newest writer and newest reader can miss failures in archived shards or lagging consumers. Inventory the writer versions actually present and the reader versions still supported. Compatibility is a property of each required pair, so an old reader that is deliberately retired should leave the support policy through an explicit migration rather than disappear from a test accidentally.

An Avro reader default supplies a value when an expected reader field is absent from the writer schema. It does not make that field optional when encoding against a writer schema that declares it. A nullable type, an absent field, and a default are therefore distinct concepts. Other validation systems have different semantics; for example, JSON Schema's default annotation should not be assumed to insert data during validation. Test the actual reader and validator instead of relying on a generic intuition that the word default guarantees a value appears everywhere.

A structurally valid default can still distort learning. Filling an unknown account age with zero can make missing history look like a new account; replacing a missing region with the most common region can alter cohort statistics. Record units, allowed values, null meaning, feature provenance, and semantic version alongside types. A change from seconds to milliseconds retains an integer type but changes values by a factor of one thousand. Likewise, reusing a category code for a new class may decode perfectly while silently changing labels. Contract review must include representative downstream calculations and quality checks, not only parser acceptance.

Keep immutable copies of supported writer schemas and pair them with archived files or a reliable schema identifier. Plan a rollout order, such as readers able to consume both forms before switching writers, with rollback behavior for the archive already produced. If a change cannot preserve required pairs, publish a new dataset or explicitly transform history instead of labeling an incompatible file as the old version. Test extreme numeric values and enum symbols in addition to typical samples. A parser success rate should identify which writer versions were exercised and which failures were excluded.

The probability lab illustrates limited test discovery, not schema resolution. For each separate configuration, Reset, set both success probability p and trial count n, predict the zero-discovery bar, then inspect. Interpret success as an independently selected test exposing a defect, and compare p=0.1, n=10 with p=0.1, n=30. More trials reduce the modeled chance of missing the defect, but do not establish that untested writer-reader pairs are compatible. Real test selection is rarely independent with a known discovery probability; deterministic required-pair tests remain the direct evidence for the migration contract.