Safe degradation and fallbacks

Separate the core operation from optional enrichment. Name which outputs may be omitted, approximated, delayed, or served from a cache, and make the changed behavior understandable to the caller where it matters. Define freshness and quality limits in the same way normal success is defined. A response that returns quickly with misleading content is not reliable merely because its HTTP status is successful. For AI systems, a smaller model or reduced retrieval set can change quality and safety behavior, so compare it using representative evaluations before calling it an acceptable degradation.

Authorization, privacy boundaries, and durable state invariants need explicit preservation. A cached protected response may require rechecking access or using a design whose invalidation and authorization scope are proven suitable. Revoked access should not silently become valid because the permission service is unavailable. Some operations should fail closed rather than fall back. Distinguish an eligible stale public result from an unsafe stale decision about ownership, account balance, or permissions. The fallback contract must name these exclusions instead of relying on a broad instruction to use the cache.

A fallback that works for one failed request can collapse when every caller uses it. Estimate total fallback demand under the triggering outage, including retries, cache misses, and any shared backend. Avoid a fallback that calls the same broken dependency through a different path. A local static response may be cheap but less useful; a remote secondary model may be useful but capacity constrained. Measure resource cost for both paths and cap admission so the fallback preserves a useful subset of work rather than causing another cascade.

Treat cold fallback state as a first-class test case. Data, credentials, compiled artifacts, and cache entries can expire when the path is rarely used. Exercise it under the intended outage workload and verify that essential dependencies remain reachable. A circuit breaker can stop repeated calls to an unhealthy service, but it needs bounded probes and a recovery policy. If all clients probe simultaneously, the attempted recovery can recreate overload. Use measured thresholds, controlled probes, and hysteresis rather than switching the entire population after one successful check.

Label fallback outcomes in telemetry separately from ordinary success and outright failure. Track eligible volume, actual fallback use, quality or freshness violations, cost, and user-visible completion. A global availability percentage can otherwise improve while product utility quietly worsens. Define when degraded outcomes count as good for each objective before activation. Preserve a way to disable a faulty fallback independently of the normal path; a stale cache or alternative model can have its own correctness incident while the main dependency is also unhealthy.

Recovery must restore normal behavior gradually and verify that fallback work and caches do not leave persistent effects. Test flapping dependency health, partial recovery, expired cache entries, and a failed fallback. The queue lab shows fluid backlog as arrival and service rates diverge, with a separately assumed M/M/1 mean residence readout. It does not know response quality, access rules, or cache freshness. Use it to illustrate the capacity benefit of cheaper work, then explain why those semantic constraints must be tested outside the model.