Chaos engineering and fault injection

Distinguish the injected cause from the observed symptom. Stopping a process tests crash handling, while delaying responses tests deadlines, cancellation, and queue behavior. Dropping packets can differ from rejecting connections immediately because callers wait and retry differently. Choose a fault whose semantics match a plausible failure mode, then identify which operation path it touches. A container restart does not necessarily test a zone outage, persistent data corruption, or an inaccessible control plane. State those exclusions so a successful experiment is not generalized beyond its evidence.

Establish a representative baseline before injection. Observe correct completed operations, latency distributions, backlog age, error categories, and relevant resource saturation. Include a control cohort when feasible and account for workload changes that could explain the same symptoms. A short experiment with no affected requests provides little evidence. Define a minimum useful sample and a maximum safe exposure before starting; if those requirements conflict, improve the environment or measurement rather than silently relaxing the safety boundary.

Define abort conditions using user outcomes and protective resource limits, not only the injector's status. A concrete test might stop if control-cohort errors rise, recovery queue age exceeds a threshold, or the observer loses visibility. The stop mechanism needs a tested path that does not rely entirely on the failing component. Bound fault duration with a lease or expiry, keep an authorized operator able to intervene, and verify that reverting the injection actually restores the intended system state. A command returning success is not recovery evidence.

Reduce the initial blast radius by using non-production environments that reproduce the relevant mechanism, then carefully selected production cohorts only within the organization's authorized process. Shared databases, credential systems, and network control paths can invalidate cohort isolation. Review them explicitly. Schedule experiments to avoid compounding unrelated maintenance and retain a rollback or containment plan for persistent state changes. Experiments involving irreversible data effects require a different design from temporary latency injection; do not assume an automated stop can undo corruption.

Continue observation after removing the fault. Watch backlog drain, cache warming, replica catch-up, client retries, and any lingering divergence. A service can return healthy probes while users still wait behind recovery work. Preserve traces and configuration versions so the outcome can be reproduced and explained. If the hypothesis fails, the useful result is an identified mechanism and a concrete correction, followed by a comparable retest. Avoid changing multiple recovery controls at once unless the experiment explicitly studies their interaction.

The replica lab is a planning aid for one narrow invariant. Set n and remove k nodes, then predict when the number up falls below floor(n/2) + 1. Use that threshold to design a process-loss hypothesis, but do not infer consensus correctness, storage durability, or correlated-domain tolerance from the picture. A real experiment must observe the protected operation, its protocol state, and recovery behavior. The lab provides a clear count to challenge; the system provides the evidence.