SLOs and incident response

An SLI needs a population, a definition of good, and a measurement location. For an interactive inference endpoint, eligible requests might exclude authenticated load tests but include real requests rejected by overload. A good request might complete without a server error, begin streaming within a target, and satisfy an inter-token latency limit. Define cancellation treatment carefully: users abandoning a slow response should not conveniently disappear from the denominator. Instrument as close to user experience as practical.

Quality and operational availability need distinct indicators. A fluent but incorrect answer can be operationally successful under a latency SLI while failing a task-quality evaluation. Track versioned evaluation results and safety requirements separately rather than claiming a latency objective proves answer quality. Slice operational indicators by region, tenant class, request size, and model version to find concentrated harm, while protecting low-volume users from noisy statistics and avoiding labels that create unbounded monitoring cardinality.

An objective below 100% permits a bounded amount of failure. The budget turns that allowance into a policy discussion: what engineering changes when reliability is being consumed too quickly? Teams might slow risky rollouts, prioritize capacity, or fix a dominant failure class. Agree on the policy with product and operations in advance. Budget exhaustion is evidence for action under that policy, not a universal instruction to stop every kind of work regardless of cause or urgency.

Burn rate compares the observed bad-event fraction with the allowed fraction. A burn of ten consumes budget ten times as fast as uniform consumption would. Multiwindow alerts can distinguish a severe current incident from a noisy short spike or a slow sustained leak. Match denominator definitions and inspect traffic changes: a request-weighted budget is consumed faster during heavy traffic even if the wall-clock duration is short. Protect against missing telemetry being interpreted as zero errors.

Tail latency grows sharply near saturation. In an ideal stable M/M/1 queue, mean residence time is $1/(\mu-\lambda)$, not merely service time $1/\mu$. Real inference has variable request lengths, batching, and bursty arrivals, so that formula is an intuition with explicit assumptions, not a production percentile predictor. Measure histogram-based percentiles or distributions; averaging p99 values from instances does not generally produce the fleet p99. Distinguish queueing, prefill, decode, and dependency time when investigating latency.

During an incident, assign coordination and communication roles, establish a timeline, and choose a reversible mitigation tied to a hypothesis. Roll back a suspect release, limit expensive requests, or move traffic only if the receiving capacity is ready. Preserve evidence without dumping secrets into logs. After recovery, examine contributing conditions, detection gaps, and the actions that would reduce recurrence or blast radius. An action item needs an owner and a measurable outcome; blaming the last person who deployed explains little about system resilience.