advanced26 min studyLesson 391 in Engineering

Metric types encode the question you can answer

Choose counters, gauges, and distributions with correct reset, time-window, and aggregation semantics before constructing an operational query.

On this page

What you’ll understand

  • Distinguish accumulating events from instantaneous state and observed distributions.
  • Explain why reset-aware counter rates should be calculated before cross-instance aggregation.
  • Aggregate means and ratios using their underlying counts rather than averaging summaries blindly.

A metric is a measurement contract before it is a line on a chart. The contract says what events or state it observes, which units it uses, and how values can be combined across time and instances. Choosing a familiar instrument name is insufficient if those semantics are wrong. We will derive a few aggregation rules so a dashboard can answer a concrete question without quietly changing its population or confusing a restart with negative work.

Accumulate events, observe state, or record a distribution

A counter accumulates a nonnegative quantity such as completed requests or bytes sent. Its ordinary trajectory rises, but process restarts can reset it. A gauge records a state that can increase or decrease, such as current queue depth or available memory. A distribution records many observations, retaining enough information for questions beyond a single total. For nonnegative durations, a count and accumulated duration can produce a mean; histogram buckets additionally preserve coarse shape. None of these instruments automatically states where an observation begins or which requests qualify.

Keep units and event boundaries explicit. A count of request attempts differs from unique user operations, and a server-side duration differs from client-perceived waiting. A gauge sampled every fifteen seconds can miss a short spike between scrapes. A counter sampled at the same frequency can preserve accumulated event count across that interval while losing timing detail, assuming the process and collection path preserve it. Instrument choice therefore trades which information survives aggregation; exporting a final average alone discards the weights needed to combine instances correctly.

r=C(t2)−C(t1)t2−t1r=\frac{C(t_2)-C(t_1)}{t_2-t_1}
For one counter with no reset or lost accounting in the interval, C counts events and t is seconds, giving average events per second. This simple endpoint formula is not the full Prometheus rate algorithm, which adjusts observed resets and extrapolates over its query window.↔ Scroll wide equations horizontally.

Preserve reset evidence and aggregation weights

Prometheus rate and increase interpret counter decreases as resets and use the observed sample sequence; they cannot recover arbitrary events that disappeared before an unobserved reset. Compute a rate for each original series before summing across instances. If one counter resets while another rises, summing the raw counters first can hide the decrease and erase evidence the reset-aware operation needed. Changing instance identity, duplicate collection, and missing scrapes require their own treatment. A smooth aggregate is not proof that every underlying source was complete and counted once.

To combine latency means, sum compatible duration totals and divide by summed observation counts. Averaging instance averages gives equal weight to each instance rather than each request, which is a different question. The same principle applies to success ratios: a quiet unhealthy instance and a busy healthy instance do not contribute equally to an event-weighted service ratio. Keep windows, units, and eligible populations aligned. Do not sum gauges blindly either: adding independent queue lengths can be meaningful, while adding replica copies of one logical value can double-count it.

x‾pooled=∑jSj∑jNj(∑jNj>0)\overline{x}_{\rm pooled}=\frac{\sum_j S_j}{\sum_j N_j}\qquad\left(\sum_jN_j>0\right)
S_j is the sum of compatible observations in one common window and N_j their count. If observations are durations in seconds, the result is seconds per observation. The denominator must be nonzero, and duplicate or overlapping populations invalidate this pooling interpretation.↔ Scroll wide equations horizontally.

Test the query against known event histories

Build fixtures containing a restart, an idle instance, a missing scrape, and unequal request counts. Predict the desired result from the event history, then compare the metric query. Decide how to represent missing or stale evidence rather than filling every gap with zero. OpenTelemetry additionally distinguishes delta and cumulative temporality and monotonic from nonmonotonic sums; exporting or converting them requires the matching data-model semantics. A collector that changes temporality needs state and reset handling, not simply a different label on the same numeric samples.

The queue lab provides a gauge example only. Reset, set both arrival λ=1 and service μ=3, predict that current backlog falls from four to zero in two seconds, then Play or Step. For a separate growing-state case, Reset and re-enter both λ=4 and μ=2, predict growth, then inspect. Rate edits retain elapsed time and recalculate from the initial four; they do not switch a live accumulated queue. The lab does not expose a request counter or implement rate queries. Its limited purpose is to show why a falling state value is not a negative event count.

Pool request latency rather than instance averages

In the same complete window, instance A observes 100 requests totaling ten seconds of latency. Instance B observes 900 requests totaling 270 seconds. No request is counted twice.

  1. A's mean is 10/100 = 0.1 seconds; B's is 270/900 = 0.3 seconds. Their unweighted mean would be 0.2 seconds per instance.

  2. The combined observation count is 100 + 900 = 1,000 and the combined duration total is 10 + 270 = 280 seconds.

  3. The request-weighted mean is 280/1,000 = 0.28 seconds, or 280 milliseconds. B contributes nine times as many requests, which the equal-instance average discarded.

The result

The service's event-weighted mean is 280 ms under the stated complete, disjoint population. A 200 ms average would answer a different instance-weighted question.

Explore the model

Interactive lab

Interactive lab · queue

Current queue depth is a gauge

Reset, set both λ=1 and μ=3, predict drainage from four to zero in two seconds, then Play or Step. Reset and re-enter both λ=4 and μ=2 for a separate growing backlog. Edits retain elapsed time and recompute from initial four. This fluid state model does not generate counters, scrapes, or reset-aware metric queries.

Current queue depth is a gaugeFluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Current readings: Fluid backlog: 4; Net rate · λ − μ: −1 /s; M/M/1 mean residence: 1 s. All parameters are adjustable with the labeled controls below.02.557.51001.534.56Elapsed time · sFluid backlogλ < μ
Fluid backlogCurrent time
Fluid backlog
4
Net rate · λ − μ
−1 /s
M/M/1 mean residence
1 s
t = 0 s

Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA state gauge can decrease without implying negative completed work; choose the instrument and aggregation for the intended question.

Check your understanding

Several independent instance counters can restart at different times. Which ordering preserves the evidence needed for a reset-aware total event rate?

Try it yourself

One instance records twenty requests with total latency four seconds; another records eighty with total latency eight seconds. Compute the pooled mean and compare it with the unweighted mean of the two instance means.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Combine totals and counts before division, then calculate the other weighting separately.

Compare with the solution

The pooled request mean is (4 + 8)/(20 + 80) = 0.12 seconds. Instance means are 0.2 and 0.1 seconds, whose equal-instance average is 0.15 seconds. Both are mathematically defined but describe different weighting; the request-weighted service question needs 120 ms here.

Take it with you

  • Counters, gauges, and distributions preserve different information about events and state.
  • Apply reset-aware operations to original counters before aggregating across instances.
  • Combine compatible totals and counts to preserve the intended population weighting.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.