Production readiness and runbooks

Start from the service contract, dependency graph, and expected workload. For each material risk, identify the evidence needed before launch: a representative load test, a rollback exercise, an access check, or a restoration report. Record the version, configuration, date, and limitations of the evidence. A capacity number from a different model size or request mix is not proof for this deployment. Readiness is specific to a change and environment; it is not a permanent certification earned once when the service was small.

Define ownership of alerts, quotas, data, and dependencies, including out-of-hours contact paths and escalation. Verify that an operator can obtain required access without relying entirely on the failing service. Name unresolved risks with a mitigation, owner, and review date rather than hiding them behind a passed aggregate score. A weighted checklist can help prioritize review, but a high numerical score must not compensate for an unresolved catastrophic invariant such as duplicate financial effects or an untested destructive migration rollback.

Begin a runbook with the symptom, affected service, prerequisites, and immediate containment options. Provide commands or observations that distinguish likely causes, explain expected output, and state what to do when output differs. Mark destructive operations, their scope, and required preconditions clearly. Prefer a small decision tree over a long page of possible commands. An operator should know when to stop and escalate rather than improvising a new recovery protocol because a step failed.

Each action needs a verification step at the user boundary. Restarting a process may restore a health probe while a queue remains blocked or state becomes inconsistent. Include rollback or containment conditions and specify how to preserve diagnostic evidence before changing state when feasible. Keep runbooks accessible during identity, network, and documentation outages using an approved independent path. Remove stale commands through periodic rehearsals with someone other than the author; a fresh operator exposes assumptions that the original developer may no longer notice.

A launch plan should include staged exposure, absolute user-outcome limits, comparison with a suitable control, and a tested way to stop expansion. Comparing only candidate and control can miss a shared dependency failure affecting both. Observe enough representative traffic to challenge the risky behavior, and acknowledge that a small canary can miss rare cases or scale-dependent contention. For state changes, verify compatibility between old and new versions before relying on binary rollback; reversing code does not necessarily reverse a schema migration.

After the rehearsal, update the evidence and runbook from what actually happened. Measure detection, decision, action, and verification separately to find the real delay. The queue lab can make a launch-recovery claim concrete: when service rate falls below arrival rate, backlog grows; after rollback, spare capacity determines drain time. Its deterministic fluid model and separate M/M/1 readout do not evaluate release quality. Use it to expose why an apparently quick rollback may still leave users waiting and why recovery headroom belongs in readiness evidence.