advanced24 min studyLesson 434 in Engineering

Compare rebuilt bytes and explain every difference

Use defined artifact scope and independent reconstruction to investigate reproducibility without mistaking identical output for safe behavior.

On this page

What you’ll understand

  • Define the outputs and environment covered by a reproducibility claim.
  • Distinguish fresh reconstruction from cache retrieval and semantic testing.
  • Investigate byte differences at their producing step.

Two builders produce inference executables that pass the same tests but have different digests. They may differ only in an embedded timestamp, or one might contain an unauthorized change. Tests alone do not resolve that question. Reproducibility asks whether the defined source, instructions, and environment permit reconstruction of identical specified output bytes. Establishing that correspondence makes unexplained variation visible; it does not prove the program implements the right model or that a shared compiler is trustworthy.

Define what must match before comparing

Choose the actual release artifacts under examination: a native wheel, an inference executable, a filesystem image, or another explicit payload. Build logs and external signatures may be outside that comparison if the claim says so. Do not quietly exclude a differing executable because its tests pass. Record the source revision, build instructions, target, toolchain, dependencies, and relevant configuration. A comparison across different declared targets can legitimately differ, but then it is not evidence that one fixed target was reconstructed. The scope must be precise enough for another builder to attempt the same reconstruction.

Compare complete specified bytes directly or through an appropriate cryptographic digest with a trusted expected value. Digest equality is practical content-identity evidence under the algorithm's assumptions; a digest copied from the same untrusted download does not independently approve anything. Preserve comparison results and identities alongside the input record. If one output differs, locate its structure and producer before assigning a cause. An archive header, code section, embedded path, or model table points to different next checks. A single aggregate match percentage cannot authorize an unexplained difference in a critical output.

∀j∈{1,…,k},H(Aj(1))=H(Aj(2))\forall j\in\{1,\ldots,k\},\quad H(A_j^{(1)})=H(A_j^{(2)})
For k specified artifacts, matching corresponding byte digests provides comparison evidence under the chosen hash assumptions. The artifacts, hash algorithm, inputs, target, and environments must be identified; this says nothing directly about semantic correctness.↔ Scroll wide equations horizontally.

Remove incidental variation at its source

Timestamps, archive ordering, filesystem enumeration, absolute build paths, locale, and random choices can change bytes without an intended source change. A documented timestamp convention such as SOURCE_DATE_EPOCH can help only when the relevant tools actually honor it. Check which generator or packaging stage emitted each variable field. Fix the producing process and rebuild both outputs, rather than merely deleting inconvenient differences after comparison. If normalization is intentionally part of artifact production, make it a versioned build step with a declared scope, so the compared artifact remains the artifact distributed to users.

Reproducible outputs can still be wrong in exactly the same way. A deterministic parser defect or harmful source modification will reproduce if both builders use it. Independent rebuilders offer stronger evidence about correspondence when they do not simply reuse the original output or share every compromised authority. Yet two organizations using the same corrupted compiler can agree perfectly. Separate the question of byte correspondence from source review, toolchain trust, behavioral tests, and isolation. Record the common inputs and trust boundaries instead of calling two matching executions independent just because they ran on different hosts.

R=MN,0≤M≤N,N>0R=\frac{M}{N},\qquad 0\le M\le N,\quad N>0
R is the dimensionless matched-artifact fraction for an explicitly enumerated nonempty set of N outputs. It is a diagnostic count, not a probability of safety or permission to ignore the remaining N−M mismatches.↔ Scroll wide equations horizontally.

A cache hit is a different observation

An output cache may speed normal development while providing no new reconstruction evidence: returning yesterday's stored artifact will naturally return yesterday's bytes. Force the relevant actions to execute in a controlled fresh environment when testing the reconstruction claim. Check that caches are keyed by complete influencing inputs and that their content integrity and writers are governed. In an AI toolchain, a cached generated kernel can mask an undeclared toolkit dependency. Retain enough logs and input identities to distinguish a genuine rebuild from retrieval, and investigate discrepancies before issuing the release's verification statement.

The cache lab makes reuse visible without pretending to compile software. For each separate run, Reset, set both capacity C and stride s, predict the initial hit and miss counts, then inspect. At C=4, s=4 the sixteen requests repeat four addresses: four misses are followed by twelve hits. At C=2, s=4 none of those requests hit. These are exact results for one empty fully associative LRU cache and its synthetic address stream. They illustrate that successful reuse and freshly executing an action answer different questions; they do not measure a build cache or establish reproducibility.

Do not hide the last differing executable

A fictional release specifies 100 artifacts for comparison. Initially 97 rebuilt artifacts match. Investigation finds two mismatches caused by a timestamp generator and one unexplained mismatch in a native kernel binary.

  1. The initial matched fraction is 97/100=97%. There are three mismatches to investigate, regardless of their sizes.

  2. Correct the timestamp generator as a declared build change and rerun both controlled builds. Assume those two artifacts now match:99 of 100 match, or 99%.

  3. The remaining kernel mismatch still prevents a complete reproducibility claim for the specified set. Investigate its inputs and generating action; neither the higher fraction nor passing model tests explains the differing bytes.

The result

After the stated repair, 99 artifacts match and one remains unresolved. Report that boundary honestly instead of publishing a claim that all specified release artifacts reproduce.

Explore the model

Interactive lab

Interactive lab · cache

Distinguish fresh work from returned outputs

For each view: Reset, set both C and stride s=4, predict, then inspect the initial sixteen requests. C=4 gives 12 hits and 4 misses; C=2 gives 0 hits and 16 misses. Play or Step only after predicting. This exact toy LRU trace is not a build-cache benchmark or reproducibility verifier.

Distinguish fresh work from returned outputsA fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Current readings: Hits / requests: 0 / 16; Hit rate: 0%; Address-stream period: 16 addresses. All parameters are adjustable with the labeled controls below.Fully associative LRU cache after 16 deterministic requests: 0 hits and 16 misses.Most recent 16 requestsaddress = i × 3 mod 160M3M6M9M12M15M2M5M8M11M14M1M4M7M10M13MCurrent cache order · oldest → newest471013Latest address 13 · miss: inserted as newestToy address stream · period 16 · starts empty
Cached addressMiss / evictionMost recent request
Hits / requests
0 / 16
Hit rate
0%
Address-stream period
16 addresses
t = 0 s

A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA returned cached result can match without executing its producing action. Reconstruction evidence must identify which actions actually ran.

Check your understanding

Two rebuilds have identical release bytes. Which conclusion follows?

Try it yourself

A specified set contains 24 artifacts. Twenty-two match and two differ only in packaging timestamps. May the current distributed set be reported as fully byte-reproducible? Compute its current match fraction.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Distinguish an identified cause from an applied and reverified build repair.

Compare with the solution

The current fraction is 22/24≈91.67%. Identifying the timestamp cause is useful, but the current bytes still differ. Fix the packaging stage and rebuild and compare the specified distributed artifacts before claiming that all 24 reproduce.

Take it with you

  • Define the exact artifact set and reconstruction context.
  • Repair and reverify byte differences rather than infer safety from a percentage.
  • Reproducibility, cache reuse, semantic correctness, and toolchain trust are separate claims.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.