Metric types encode the question you can answer
Choose counters, gauges, and distributions with correct reset, time-window, and aggregation semantics before constructing an operational query.
On this page
The question for today
What you’ll understand
- Distinguish accumulating events from instantaneous state and observed distributions.
- Explain why reset-aware counter rates should be calculated before cross-instance aggregation.
- Aggregate means and ratios using their underlying counts rather than averaging summaries blindly.
A metric is a measurement contract before it is a line on a chart. The contract says what events or state it observes, which units it uses, and how values can be combined across time and instances. Choosing a familiar instrument name is insufficient if those semantics are wrong. We will derive a few aggregation rules so a dashboard can answer a concrete question without quietly changing its population or confusing a restart with negative work.
Accumulate events, observe state, or record a distribution
A counter accumulates a nonnegative quantity such as completed requests or bytes sent. Its ordinary trajectory rises, but process restarts can reset it. A gauge records a state that can increase or decrease, such as current queue depth or available memory. A distribution records many observations, retaining enough information for questions beyond a single total. For nonnegative durations, a count and accumulated duration can produce a mean; histogram buckets additionally preserve coarse shape. None of these instruments automatically states where an observation begins or which requests qualify.
Keep units and event boundaries explicit. A count of request attempts differs from unique user operations, and a server-side duration differs from client-perceived waiting. A gauge sampled every fifteen seconds can miss a short spike between scrapes. A counter sampled at the same frequency can preserve accumulated event count across that interval while losing timing detail, assuming the process and collection path preserve it. Instrument choice therefore trades which information survives aggregation; exporting a final average alone discards the weights needed to combine instances correctly.
Preserve reset evidence and aggregation weights
Prometheus rate and increase interpret counter decreases as resets and use the observed sample sequence; they cannot recover arbitrary events that disappeared before an unobserved reset. Compute a rate for each original series before summing across instances. If one counter resets while another rises, summing the raw counters first can hide the decrease and erase evidence the reset-aware operation needed. Changing instance identity, duplicate collection, and missing scrapes require their own treatment. A smooth aggregate is not proof that every underlying source was complete and counted once.
To combine latency means, sum compatible duration totals and divide by summed observation counts. Averaging instance averages gives equal weight to each instance rather than each request, which is a different question. The same principle applies to success ratios: a quiet unhealthy instance and a busy healthy instance do not contribute equally to an event-weighted service ratio. Keep windows, units, and eligible populations aligned. Do not sum gauges blindly either: adding independent queue lengths can be meaningful, while adding replica copies of one logical value can double-count it.
Test the query against known event histories
Build fixtures containing a restart, an idle instance, a missing scrape, and unequal request counts. Predict the desired result from the event history, then compare the metric query. Decide how to represent missing or stale evidence rather than filling every gap with zero. OpenTelemetry additionally distinguishes delta and cumulative temporality and monotonic from nonmonotonic sums; exporting or converting them requires the matching data-model semantics. A collector that changes temporality needs state and reset handling, not simply a different label on the same numeric samples.
The queue lab provides a gauge example only. Reset, set both arrival λ=1 and service μ=3, predict that current backlog falls from four to zero in two seconds, then Play or Step. For a separate growing-state case, Reset and re-enter both λ=4 and μ=2, predict growth, then inspect. Rate edits retain elapsed time and recalculate from the initial four; they do not switch a live accumulated queue. The lab does not expose a request counter or implement rate queries. Its limited purpose is to show why a falling state value is not a negative event count.
Work it through
Pool request latency rather than instance averages
In the same complete window, instance A observes 100 requests totaling ten seconds of latency. Instance B observes 900 requests totaling 270 seconds. No request is counted twice.
A's mean is 10/100 = 0.1 seconds; B's is 270/900 = 0.3 seconds. Their unweighted mean would be 0.2 seconds per instance.
The combined observation count is 100 + 900 = 1,000 and the combined duration total is 10 + 270 = 280 seconds.
The request-weighted mean is 280/1,000 = 0.28 seconds, or 280 milliseconds. B contributes nine times as many requests, which the equal-instance average discarded.
The service's event-weighted mean is 280 ms under the stated complete, disjoint population. A 200 ms average would answer a different instance-weighted question.
The living notebook
Explore the model
Interactive lab · queue
Current queue depth is a gauge
Reset, set both λ=1 and μ=3, predict drainage from four to zero in two seconds, then Play or Step. Reset and re-enter both λ=4 and μ=2 for a separate growing backlog. Edits retain elapsed time and recompute from initial four. This fluid state model does not generate counters, scrapes, or reset-aware metric queries.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA state gauge can decrease without implying negative completed work; choose the instrument and aggregation for the intended question.
Pause & reason
Check your understanding
Your turn
Try it yourself
One instance records twenty requests with total latency four seconds; another records eighty with total latency eight seconds. Compute the pooled mean and compare it with the unweighted mean of the two instance means.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Combine totals and counts before division, then calculate the other weighting separately.
Compare with the solution
The pooled request mean is (4 + 8)/(20 + 80) = 0.12 seconds. Instance means are 0.2 and 0.1 seconds, whose equal-instance average is 0.15 seconds. Both are mathematically defined but describe different weighting; the request-weighted service question needs 120 ms here.
Keep the idea
Take it with you
- Counters, gauges, and distributions preserve different information about events and state.
- Apply reset-aware operations to original counters before aggregating across instances.
- Combine compatible totals and counts to preserve the intended population weighting.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.