Failure models and dependency graphs

For a document lookup, draw the frontend, authorization decision, metadata store, object store, and any transformation needed before the response can be useful. Mark which edges must succeed during this particular operation. A cached public thumbnail may avoid the transformation service, whereas downloading a private original cannot avoid authorization. The resulting graph describes conditional execution paths rather than a list of every service owned by the organization. Annotate request volume and resource cost on those paths.

Define a failure model explicitly: process crash, delayed response, network partition, stale read, corrupted data, or arbitrary behavior. A design that survives one process crash may still accept conflicting writes during a partition. Availability also depends on the measurement boundary: a successful internal RPC is not proof that the user received a correct document. Record the observation point, time interval, expected workload, and correctness conditions before attaching probabilities to nodes. Otherwise two teams can report excellent availability while measuring different outcomes.

Two independent alternatives can reduce the probability of losing the operation, but duplication alone does not create independence. Model a shared failure event first, then condition the remaining model on its absence. Examples include a shared encryption-key service, an invalid global configuration, and an operator action applied to both copies. Classifying racks or regions as distinct is evidence about some failure causes; it does not eliminate shared software defects, permissions, or dependencies. A dependency graph should therefore include both physical domains and operational control paths.

A minimal cut set is a set of failures sufficient to make the modeled operation unavailable for which no proper subset is sufficient; it need not have the smallest possible cardinality. Enumerating small cut sets often exposes a surprising singleton, such as a configuration download needed during every restart. Prioritize removing that singleton or giving the operation a tested local capability. Do not assume failure events are mutually exclusive when adding probabilities. The union bound gives a conservative upper bound without independence, while measured joint incidents can support a better model if the sample and uncertainty are adequate.

Compare incident traces with the predicted cut sets. Were replicas actually restarted together? Did traffic shift exceed spare capacity? Did restoring credentials require the broken service? Keep uncertainties visible rather than estimating rare-event probabilities from a quiet month. A useful engineering output is a testable claim: one storage process can be removed during a specified workload while successful lookup latency stays within its boundary. That statement is narrower, more useful, and easier to falsify than saying the architecture is highly available.

Use the replica lab to remove nodes and inspect how many remain relative to a majority. Then redraw the imagined placement so one domain removes several nodes together. The controls only show counts, so correlated domains remain part of your written analysis. A majority being up is necessary for some quorum protocols, but this lab does not implement voting, log agreement, fencing, or durable storage. It cannot prove linearizability or determine whether the surviving nodes have the newest data. Connect every resilience claim to its protocol and evidence.