advanced28 min studyLesson 394 in Engineering

Trace context connects observations without inventing a global clock

Propagate trace and span identity, use links for non-tree relationships, and interpret durations without equating a partial trace with complete causality.

Before you begin

On this page

The question for today

What you’ll understand

Distinguish trace identity, span identity, parent relationships, and links.
Explain why overlapping spans and unsynchronized clocks defeat naive duration sums.
Identify missing instrumentation and collection as limits on causal conclusions.

A trace organizes selected observations of work across components. Context propagation lets a downstream operation refer to the work that initiated it, while spans record chosen boundaries and events. This makes production behavior easier to investigate, but the result is still an instrumented, sampled, collected view. It is not a complete execution history, a synchronized global timeline, or an authorization token. We will preserve those distinctions when interpreting a plausible-looking waterfall.

Propagate identity at execution boundaries

A trace ID groups related spans, and a span ID identifies a particular operation within that trace. A parent span relationship expresses the instrumentation's chosen parent context; OpenTelemetry links can record additional relationships such as a batch consuming messages from several producers. One parent pointer cannot encode every dependency in such a fan-in. Carry context across supported network, queue, and asynchronous execution boundaries deliberately. If a task loses its active context or reuses an unrelated one, the displayed topology can omit or misattribute work without the underlying request having disappeared.

W3C Trace Context defines interoperable propagation fields, including traceparent and tracestate, rather than the application's full span database. Validate incoming context and apply the system's trust-boundary policy; correlation identifiers are not credentials and a remote sampled flag is not permission to spend unlimited resources. Do not place personal data inside these fields. Span names and attributes need similarly deliberate scope. Stable operation names support analysis, while raw identifiers belong only where the data policy and instrumentation contract permit their collection.

d_i=t^{\rm local}_{i,\rm end}-t^{\rm local}_{i,\rm start}

A span's elapsed duration d_i is seconds when both boundaries use a suitable consistent local timing source. Cross-host wall-clock subtraction requires additional clock assumptions; valid trace identifiers do not synchronize those clocks or turn every span boundary into a CPU-time measurement.↔ Scroll wide equations horizontally.

A waterfall is not additive resource accounting

Concurrent child spans can overlap, so summing their elapsed durations can exceed their parent's elapsed duration. Nested durations can also double-count the same waiting interval. To discuss coverage of a parent window, take the union of relevant child intervals under a justified common time coordinate rather than simply adding them. Even uncovered parent wall time is not automatically CPU time: it can include uninstrumented waits, scheduling, queueing, or missing spans. A CPU profiler and an event trace answer different questions.

Across hosts, timestamp offsets and drift can make a downstream operation appear to start before the displayed upstream event. Known message relationships can establish a logical dependency without telling you the exact one-way delay from those timestamps. Instrumentation may also let asynchronous work outlive the parent operation's measured boundary. Parent edges and links document selected relationships, not a universal proof of complete causality. Preserve clock uncertainty and trace completeness indicators when deciding whether a surprising picture is an application behavior, instrumentation bug, or display artifact.

D_{\rm covered}=\left|\bigcup_i\left(I_i\cap I_{\rm parent}\right)\right|\le\sum_i|I_i\cap I_{\rm parent}|

I_i are relevant child time intervals and vertical bars denote elapsed interval length in seconds in one justified time coordinate. Union length avoids overlap double counting. It measures instrumented time coverage, not causal completeness, CPU consumption, or all work performed.↔ Scroll wide equations horizontally.

Check propagation and completeness before assigning causes

Test traces for a simple request, retries, parallel fan-out, queued work, and batched fan-in. Verify the expected contexts and links rather than merely confirming that spans arrive. Track sampling, export failure, late arrival, and instrumentation coverage because each can remove different pieces. If a database span is absent, distinguish no database call from an uninstrumented, unsampled, or lost observation. Correlate application outcomes and independent metrics where possible. A trace can suggest a hypothesis that a controlled reproduction or additional evidence then tests.

The probability lab illustrates incomplete observation under a narrow hypothetical model. Reset, set both p=0.5 and n=20, predict ten retained observations on average, then Play or Step. Each trial means an independent observation opportunity with common retention probability, not a span relationship in a real trace. Reset and re-enter both controls before a comparison. The exact binomial bars are not sampled executions and cannot represent parent edges, links, clock offsets, or correlated instrumentation gaps. Their purpose is to remind us that absence in a retained view need not mean absence of underlying work.

Work it through

Overlapping spans exceed the parent when added

In a deliberately common clock coordinate, a parent spans 0–200 ms. Child A spans 30–130 ms and child B spans 50–170 ms. Both are relevant operations, but neither span measures CPU time.

A lasts 100 ms and B lasts 120 ms, so their duration sum is 220 ms, exceeding the parent's 200 ms because they overlap.
Their overlap is 50–130 ms, or 80 ms. The union covers 100 + 120 − 80 = 140 ms inside the parent interval.
The parent has 200 − 140 = 60 ms outside that observed union. That is uncovered wall time, not proof of sixty milliseconds of parent CPU or proof that no other work occurred.

The result

Use interval coverage rather than naive sums, and keep the timing and instrumentation assumptions attached to the conclusion.

The living notebook

Explore the model

Interactive lab

Interactive lab · probability

Missing observation is not missing work

Reset, set both p=0.5 and n=20, predict ten retained observations on average, then Play or Step. Reset and re-enter both controls for another case. These exact independent-retention bars do not depict real spans, parent links, clock synchronization, or a complete execution graph; playback highlights outcomes rather than sampling work.

Exact probabilityHighlighted outcomeMean np

Mean · np: 5
Variance · np(1−p): 2.5
Total probability: 1

Success probability p0.5Trials n10

t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forTrace topology and timing conclusions depend on propagation, clocks, and which observations survived collection.

Pause & reason

Check your understanding

Interactive checks need JavaScript. You can still read the explanation.

Read the checkpoint explanation

A trace is a partial instrumented observation. Absence can motivate investigation, but additional evidence is needed to distinguish no work from an unobserved or lost span.

Your turn

Try it yourself

Two child intervals in one justified clock coordinate cover 10–60 ms and 40–90 ms within a 0–100 ms parent. Compute their summed duration, union coverage, and uncovered parent wall time.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Subtract the overlapping 40–60 ms interval once before comparing with the parent.

Compare with the solution

The durations are fifty milliseconds each, summing to 100 ms. Their twenty-millisecond overlap makes union coverage eighty milliseconds, from ten through ninety. The parent has twenty milliseconds outside that union. This is wall-time coverage, not an exclusive-CPU attribution or a proof that other work was absent.

Keep the idea

Take it with you

Trace/span identity, parent relationships, and links express selected instrumentation relationships.
Overlapping elapsed durations and cross-host clocks require explicit reasoning before arithmetic.
Missing spans and context gaps limit causal conclusions and call for corroborating evidence.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.