Observability: turn a symptom into a testable explanation

Metrics summarize behavior as measurements over time: request counts, error fractions, durations, or queue depth. Logs record events with context. Traces connect spans of work across an operation's path, helping locate time spent in dependencies. These signals complement one another. A count can reveal a broad change without explaining one failed request; a trace can explain one path without representing every user's experience.

Choose labels and retained context carefully. A user ID on every metric can create an enormous number of distinct time series, while raw request bodies can expose private data. Prefer bounded dimensions for aggregate metrics and controlled, appropriate context for event investigation. Correlation identifiers can connect evidence across boundaries, but missing propagation, sampling, clock differences, and broken telemetry must be considered when interpreting gaps.

An average can hide a slow minority. A latency percentile describes the value below which a specified fraction of observations falls under the chosen definition. Compare the median with higher percentiles and the request population behind them. A percentile from a quiet endpoint can be unstable, and averaging percentiles across machines does not generally produce the percentile of the combined requests. Aggregate the underlying distribution appropriately.

Traffic, latency, errors, and saturation are a useful starting set of service signals. Saturation concerns how close a resource is to an effective limit, which may involve queued work rather than just CPU utilization. A machine can have low CPU usage while waiting on storage. A growing queue may be the cause of added waiting time, or a symptom of a slower downstream service; the signals need a causal investigation.

Write a concrete observation: the high-percentile duration of successful inference requests increased after a particular release while traffic stayed roughly similar. Propose alternatives, such as larger inputs, reduced batching efficiency, a slow dependency, or resource contention. Identify a measurement that separates them. A temporal correlation suggests where to look; it does not establish that the release caused the change or explain its mechanism.

Prefer reversible, bounded investigations and preserve evidence before it disappears. Compare versions, inspect representative traces, and change one relevant variable when practical. The queue lab shows fluid backlog starting at four jobs and changing at arrival rate minus service rate, clipped at zero. Its separately labeled M/M/1 mean assumes independent Poisson arrivals, exponential service times, one server, and arrival rate below service rate. Neither is a fitted production forecast. Pause and define which jobs your real measurements count before applying Little's law to a dashboard.