Reconcile identities and values at a defined cutoff

A source of truth is authoritative for a particular fact under a contract, not necessarily for every field in every representation. A collection system might own the original event while a reviewed labeling system owns its corrected class. A feature pipeline derives values from both using a versioned transformation. Define stable keys, merge rules, tombstones, and field ownership before attempting repair. Picking whichever record has the latest wall timestamp can select a stale or incorrectly clocked update, and a larger sequence number is meaningful only within the sequence's defined scope and incarnation.

Compare equivalent populations. Pin source and target snapshots, source log positions, transformation version, and the policy cutoff used for membership. A stream target may legitimately lag the source, while an intentionally filtered training set should not contain every source key. Build the expected target set by applying the specified rules to the authoritative source state, then compare with the actual target. Reconcile corrections and deletions as well as additions. Otherwise a pipeline that ingests all new records but never removes withdrawn ones can appear healthy on an append-only count dashboard.

A watermark often expresses an estimate or contract about event-time progress, not an unconditional certificate that no older event can arrive. Its meaning depends on the source and late-data policy. A maximum observed sequence can also hide gaps: seeing 100 does not prove that 99 was applied. For partitioned logs, retain the relevant contiguous position per partition and account for transactional or cross-stream relationships. If the available evidence cannot establish a common completed boundary, label the comparison provisional and specify when late data or a later snapshot will trigger another pass.

After aligning keys, compare the values covered by the transformation contract, including null semantics, units, label versions, and normalized representations. Equal sums can hide offsetting errors; equal row counts can hide replacements; equal digest summaries depend on their scope and collision assumptions. Partition-level checksums can narrow an investigation but should lead to appropriate record-level verification for the claim being made. Track duplicates separately before converting rows into key sets, because a set comparison silently removes multiplicity that can change training weights. State whether ordering matters for the downstream consumer.

A repair should be reproducible, scoped, and idempotent: insert the missing expected version, remove an extra version under the policy, or recompute a mismatched value from pinned authoritative inputs. Record the repair run and verify the result at a new clearly defined boundary. Replaying an old source without the current required suppression or correction rules can restore the very data the reconciliation was meant to remove. Preserve evidence of what differed before overwriting it, and test that normal ingestion will not immediately reproduce the same mismatch after the repair finishes.

The probability lab shows the limits of a small spot audit. Reset, set both p=0.01 and n=30, predict the zero-defect bar, then inspect. Under independent equal-probability selection, it is 0.99 raised to the thirtieth, about 74%. The instrument does not reconcile keys, compare values, or model finite-population sampling without replacement. It explains why seeing no defect in a small convenience check cannot establish equality of two large sets. Use aligned full comparisons or a justified sampling design matched to the required confidence and defect scope.