Failure domains beyond path count

Start at each endpoint and trace links, transceivers, switches, power feeds, and management or routing dependencies. Mark where paths meet. Diversity at one layer can hide a shared lower-layer resource, such as two logical interfaces behind one adapter or two switches on one power domain. Software rollout and configuration can also fail several devices together. A useful failure-domain map includes operational dependencies rather than only the forwarding diagram shown during normal conditions.

Define the event being modeled: loss of reachability for a flow, loss of all job participants, degraded throughput, or a deadline miss. Those events have different dependency structures. An apparently independent link-failure probability cannot be substituted for the probability that an entire distributed job misses its iteration target. State the observation window and conditional assumptions before combining probabilities. Historical incident correlations can challenge the model, but finite data cannot prove future independence.

Consider a common event that removes both paths, plus separate path failures conditional on that event not occurring. Count the common case once, then add the conditional joint failure case. Multiplying two unconditional small percentages can miss a much larger shared risk. The model can still be incomplete if supposedly separate failures share repair resources or environmental conditions. Present it as a declared scenario and sensitivity analysis rather than a measured reliability guarantee.

Failure recovery changes traffic placement. If one of two equal paths disappears, existing work may hash onto the remaining path, reconnect, or wait for a timeout. Routing convergence and transport recovery are different intervals. Even after convergence, the residual capacity can be smaller than offered demand. Check post-failure queue growth and application deadlines, and include maintenance states in which redundancy is already reduced. A normal-state capacity test does not establish an acceptable failure budget.

Use controlled fault tests with documented scope, observation, rollback, and useful-work checks. Disable or degrade a chosen path and observe convergence, retries, correctness, and time to steady completion. Test a shared dependency separately from a single link. A partially failing or high-error link can be more disruptive than a clean removal because traffic continues selecting it. Correlate application symptoms with both active probes and device telemetry rather than relying on one link-state bit.

The reliability lab counts up replicas and a majority threshold, not network path diversity. Reset and set both n=5, k=2; predict three available and a majority of three, then inspect. Reset and re-set n=5, k=3; predict only two available. Play changes illustration state, not failure probabilities or routing. The lesson is limited: counting reachable resources does not prove independent failure domains, adequate bandwidth, or replica consistency. Those properties require a separate dependency and protocol argument.