Tests as evidence, reproducibility as a method

A unit test can check a pure transformation without starting a database. An integration test can check whether a real database enforces a uniqueness rule. An end-to-end test can verify that a user submits a valid request and sees the resulting state. These names describe boundaries, not a ranking of virtue. Choose the boundary at which the requirement can actually fail, balancing speed, realism, and diagnostic clarity.

A test that computes its expected answer with the same logic as the implementation can preserve the same mistake twice. Derive examples independently, use known results, and include boundary cases. For a normalization function, test empty input policy, invalid values, and a property such as a valid output sum. For an API, check both accepted requests and the documented behavior for malformed or repeated submissions.

A reproducible computation needs recorded inputs, code, dependencies, configuration, and relevant execution conditions. Setting a seed controls a particular pseudorandom sequence, but parallel scheduling, nondeterministic operations, library versions, and hardware can still change results. Separate exact replay from statistical reproducibility: repeated training may produce different parameters while still supporting the same conclusion within a measured range of variation.

Keep test data small enough to understand and representative enough to reveal the behavior being checked. Do not put private production records or secrets into fixtures. For an experiment, record data splits and preprocessing as carefully as model code; otherwise a result can change without an obvious source edit. When a test fails, preserve the minimal relevant evidence and identify whether the failure concerns code, data, environment, or the test's own assumptions.

An intermittent failure can expose a race, an external dependency, or a timing assumption in the test. Rerunning until green hides evidence if the underlying cause remains. Isolate dependencies where appropriate, use explicit readiness conditions instead of arbitrary delays, and retain failing traces. If the product requirement involves a real integration, preserve a meaningful integration check rather than replacing every dependency with a convenient mock.

The lab shows an exact binomial distribution for a chosen success probability and independent trial count. It does not run tests or infer the probability from data. Label an event consistently as success, vary the assumed probability, and observe how the predicted count distribution changes. Correlated test cases or shared blind spots violate the independence model, so do not interpret its bars as proof of coverage. Before adding more tests, ask which unverified requirement the new test would challenge and what a failure would teach you.