Multi-region failover

Write down which traffic survives a regional loss, where it goes, and which features can be degraded. A destination's normal utilization is not the same as its failover headroom. Include cold caches, connection storms, replication catch-up, request mix, and autoscaling delays in a measured failover workload. Spare compute without sufficient database throughput or network quota may be unusable. If capacity must be acquired during the incident, test that dependency and acknowledge the risk of simultaneous demand from other users of the same infrastructure.

Active-active deployment spreads ordinary traffic, while active-passive deployment keeps a destination idle or partly warm. Neither label specifies the data consistency model or the actual spare capacity. A two-region design running both regions near saturation cannot absorb losing one without shedding work or adding capacity. Predefine which request classes are admitted after a failure, and ensure the destination can reject excess work cheaply. Regional failover should not convert a contained outage into global overload through unbounded retries and redirected traffic.

A region can be unreachable from an observer while still processing requests from other clients. Promoting a new writer merely because a probe timed out risks conflicting writers. Use the datastore's supported promotion and fencing protocol to determine authority, and verify the state that will be served. A fencing mechanism must be enforced at the protected resource; an old process remembering that it once held a lease is insufficient. Manual intervention must follow the same safety model rather than bypassing it under pressure.

For asynchronous replication, record what has reached the recovery destination and how that relates to acknowledged user writes. A measured lag sample does not guarantee a maximum loss bound during every failure. For synchronous replication, determine which acknowledgements define durability and what happens when the remote path is unavailable. Changing a quorum membership or forcing a promotion is a protocol operation, not a count adjustment. Reconcile in-flight operations, idempotency records, and external side effects before treating the destination as a coherent replacement.

Routing convergence depends on client caches, connection reuse, health checks, and the routing mechanism. Do not equate a DNS TTL with a guaranteed global switch time. Observe successful operations from relevant client locations and retain a way to pause traffic movement if the destination saturates. Verify that administrative access, secrets, certificates, deployment images, and telemetry do not depend entirely on the failed region. A runbook that requires downloading credentials from the unreachable environment contains a circular recovery dependency.

Failback deserves the same care as failover. Restore the failed region as a follower or otherwise reconcile it using the supported protocol; do not simply restart the former writer and redirect users. Confirm replication convergence, capacity, and rollback conditions before moving traffic in stages. In the replica lab, compare losing one node with losing the several nodes assigned to one region. The diagram only counts up nodes and a majority; it cannot model placement, partitions, promotion safety, or recovery data. Supply those details in the written failure plan.