advanced27 min studyLesson 397 in Engineering

Exemplars connect an aggregate to an example, not a complete population

Use contextual references to move from metrics to traces and logs while measuring link resolution and preserving each signal's selection limits.

On this page

What you’ll understand

  • Distinguish an exemplar's contextual attributes from metric series labels.
  • Trace correlation through selected measurements, retained spans, and available log records.
  • Explain why selected examples cannot replace population denominators or prove a common cause.

An aggregate tells us that a behavior became common or slow, while an example helps us inspect one concrete occurrence. Exemplars provide a bridge between those views by attaching selected measurement context, often including trace identity. The bridge is useful precisely because the full event payload is not attached to every metric sample. Its limits matter too: an exemplar is a selected example, and its referenced trace or logs can be missing, expired, or inaccessible.

Keep event context outside the series identity

A metric data point represents an aggregation over a population and time interval. An exemplar preserves context from a selected contributing measurement, such as its value, time, and associated trace/span identifiers under the SDK's rules. These attributes are not an invitation to add a trace ID label to the metric's ordinary series identity. Doing that can create a new series per trace and defeat aggregation's bounded cost. Read the actual exporter and backend support because an exemplar can be filtered, sampled, limited, or discarded along the path.

Selection policies differ. A trace-based eligibility filter can restrict which measurements are candidates, and a reservoir then chooses among eligible candidates. Do not assume that an exemplar is uniformly representative of all requests or that it is always the slowest request in its bucket. OpenMetrics and OpenTelemetry have their own representation and reservoir contracts. Treat any sampled example as evidence that this particular observed behavior occurred, with its documented provenance, rather than as an exact count of all events sharing its attributes.

fbad=NbadNeligible(Neligible>0)f_{\rm bad}=\frac{N_{\rm bad}}{N_{\rm eligible}}\quad(N_{\rm eligible}>0)
The event fraction needs the compatible aggregate population counts. An exemplar is contextual evidence attached to a selected measurement and is not a replacement numerator or denominator. Counts must use the same eligibility definition, window, and measurement boundary.↔ Scroll wide equations horizontally.

A link joins records that survived separate pipelines

Following a trace ID from a metric requires the trace to have been recorded, retained, delivered, indexed, and still available under the user's access policy. Following from that trace to logs requires log context propagation and independent collection/retention behavior. A missing destination can therefore mean many things besides an invalid ID. Preserve service, environment, time range, instrumentation scope, and identifier semantics so a UI does not query an unrelated backend or silently join identically named resources across different environments.

Time proximity alone does not establish a causal relationship. An error log near a latency spike might belong to another request or reflect a common upstream cause. A shared trace context is stronger correlation evidence for the instrumented relationship, but incomplete traces and manually attached attributes still have limits. Verify the operation and event sequence, then test the suspected mechanism with additional observations or a controlled reproduction. A dashboard link can accelerate investigation; it should not manufacture certainty by hiding how the joined records were selected.

cresolve=Nresolved referencesNattempted references(Nattempted references>0)c_{\rm resolve}=\frac{N_{\rm resolved\ references}}{N_{\rm attempted\ references}}\quad(N_{\rm attempted\ references}>0)
This measures reference-resolution coverage for a specified set of attempted exemplar links, not the service's request success rate or an unbiased population trace-retention rate. Failures can include expiry, loss, permissions, incorrect routing, or invalid references.↔ Scroll wide equations horizontally.

Test the bridge as a user-facing diagnostic path

Create a known test request that emits compatible metrics, spans, and a structured event, then follow the references through the actual data path. Exercise absent traces, old retention windows, restricted access, different environments, and late indexing. The UI should distinguish a link that resolved to no retained data from proof that no work occurred. Track resolution failures and provenance without exporting sensitive context indiscriminately. A metric can remain a valid complete count even when its attached examples are sparse, provided its own collection contract is satisfied.

The probability lab gives a hypothetical resolution model only. Reset, set both p=0.75 and n=20, predict fifteen resolvable references on average, then Play or Step. Reset and re-enter both parameters before a separate comparison. The exact binomial bars assume independent references with one common resolution probability; real expiry, backend outages, or access policies can correlate results. Playback highlights the distribution rather than resolving URLs or sampling traces. It cannot infer why an actual link failed or turn selected exemplar cases into a representative request population.

Keep aggregate reliability and diagnostic coverage separate

A complete metric window counts 10,000 eligible requests, with 9,600 satisfying a defined latency threshold. Across several collection points, twenty retained exemplar references are inspected and fifteen resolve to retained traces.

  1. The metric's bad count is 10,000 − 9,600 = 400, so its recorded bad-event fraction is 400/10,000 = four percent.

  2. The inspected reference-resolution fraction is 15/20 = 75 percent. Its denominator is attempted exemplar references, not eligible requests.

  3. The five unresolved references motivate checking retention, routing, loss, and access. They neither change the metric count to five errors nor establish that 25 percent of all request traces are missing, because exemplar selection and reference inspection may be biased.

The result

Four percent describes the complete recorded request population, while 75 percent describes this inspected diagnostic bridge. The two measurements answer different questions.

Explore the model

Interactive lab

Interactive lab · probability

Resolution coverage under a hypothetical independent model

Reset, set both p=0.75 and n=20, predict mean fifteen resolved references, then Play or Step. Reset and re-enter both controls for another case. The exact binomial model assumes independent equal-probability reference resolution; it does not follow links, apply permissions, join signals, or correct exemplar selection bias.

Resolution coverage under a hypothetical independent modelThese are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Current readings: Mean · np: 5; Variance · np(1−p): 2.5; Total probability: 1. All parameters are adjustable with the labeled controls below.03581000.070.150.220.3Successes kP(X = k)k = 0 · P = 9.77e-4
Exact probabilityHighlighted outcomeMean np
Mean · np
5
Variance · np(1−p)
2.5
Total probability
1
t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA diagnostic link's availability and a service's population reliability require different denominators and evidence.

Check your understanding

Why should a trace identifier in an exemplar generally remain separate from the metric's ordinary label set?

Try it yourself

A metric records fifty bad events among 2,000 eligible events. Separately, eight of ten inspected exemplar links resolve. Compute both fractions and explain why the unresolved links are not an event-error count.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Keep the two populations and their measurement boundaries separate.

Compare with the solution

The bad-event fraction is 50/2,000 = 2.5 percent. The inspected link-resolution fraction is 8/10 = eighty percent. Two unresolved links concern diagnostic record availability or lookup behavior, not two additional failed user events, and the selected inspected examples need not represent all requests.

Take it with you

  • Exemplars attach selected measurement context without making event IDs metric-series labels.
  • Cross-signal correlation depends on propagation, retention, access, and compatible resource identity.
  • Measure diagnostic resolution separately and preserve selection limits when interpreting examples.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.