Observe cache value and test safe bypass

Record positive fresh hits, permitted stale hits, authoritative negatives, transient-error reuse, misses, revalidations, and coalesced waits separately. A hit may still involve network and decoding cost, and a coalesced follower may wait nearly as long as a miss while avoiding extra origin work. Track end-to-end latency, age, source version where available, and origin cost. A global average can hide one tenant or key family with poor correctness or a hot-key bottleneck.

Use compatible denominators and intervals. Request hit rate, byte hit rate, and avoided origin CPU can diverge sharply. A cache server's lookup miss count may include retries or internal probes absent from the application's request count. A fill rejected by admission or a version guard is not necessarily a malfunction; its reason matters. Avoid raw-key labels that expose private data or create unbounded metric cardinality. Aggregate by deliberate nonsecret cohorts and retain sampled diagnostic traces under appropriate access controls.

If a fraction of requests bypasses a cache independently, those requests go to the origin even when they would have hit. The remaining requests retain their original miss fraction only as a planning assumption. A true cache outage can also trigger retry waves, lost coalescing, cold dependent caches, and a different request mix. Start with a controlled fraction and a stop condition based on origin saturation, error rate, and latency rather than jumping directly to complete bypass.

Shadow reads can compare cached and origin versions, but still consume origin capacity and need correct authorization, deadlines, and nonmutating semantics. Do not replay an unsafe operation merely because it is labeled a cache check. Stale comparisons need a common source-version or age interpretation; reading a lagging replica can make a newer cache look wrong. Sample discrepancies and classify their cause before automatically replacing values or broadening invalidation.

A cache-loss test is incomplete without recovery. Warming many entries simultaneously can recreate a stampede even after the origin has survived the outage. Pace fills, preserve version guards, and prioritize actually requested valuable keys rather than loading every historical key blindly. Observe eviction churn and memory pressure as the cache returns. Compare restored freshness and useful latency with the baseline, not merely whether connections reconnect and the hit counter begins increasing.

The queue lab makes the capacity assumption visible. Reset and set both λ=2, μ=3, predict declining backlog from four, then Step. For a separate higher-origin-load scenario Reset and re-set λ=4, μ=3, predict growth and Step. Rates are abstract jobs, not this lesson's measured QPS; the model has no hit ratio, bypass switch, retries, or saturation feedback. Use it to explain why a rate increase crossing service capacity needs a stop condition, then map actual measured rates to a justified admission budget.