Sampling for investigation and sampling for estimation differ

A head decision happens before the full operation outcome is available, often at span or root-trace creation. A coordinated parent-based policy can help retain a coherent related set, while incompatible decisions or later loss can still create gaps. Tail sampling waits for enough completed observations to apply policies such as keeping errors or long durations. That requires routing related data together, buffering, decision timeouts, and handling late spans. A tail sampler cannot recover spans that an upstream head sampler never recorded or exported.

Adaptive sampling changes rates to meet a volume or investigation budget. Record which policy and effective inclusion probability applied to each estimable observation; a rate configured at noon need not describe a span retained earlier or after several collection stages. Dependent and consistent sampling stages do not always combine by naively multiplying nominal configuration percentages. Follow the actual probability contract. Deterministic exclusions with zero inclusion probability make some population totals unidentifiable from the sample alone, regardless of how sophisticated a later query appears.

If failures are retained more often than successes, the raw fraction of failures among stored traces overstates their population fraction. This can be desirable for debugging but must be labeled accordingly. An inverse-probability estimator can estimate totals when inclusion probabilities are known, positive, and preserved through the relevant pipeline. A ratio of two estimated totals is not generally exactly unbiased in a finite sample. If a reliable unsampled request total is independently available, dividing an unbiased estimated failure total by that known denominator has a clearer expectation argument.

Sampling uncertainty remains even with a valid estimator. Rare events can be missed, weights can create high variance, and trace-level selection correlates spans within one trace. Confidence calculations must use the actual sampling unit and design rather than treating every retained span as an independent request. Compare derived estimates with dedicated complete counters where possible, and expose policy changes or probability metadata gaps. A selected set of interesting traces is valuable evidence about possible failures, but cannot establish how frequently all users experienced them without the missing statistical conditions.

Exercise the policy with normal requests, rare failures, long-running traces, bursts, missing children, and late arrivals. Measure incoming and retained volume, buffering, dropped data, and the probability information available to queries. Verify that an upstream sampling flag or trace ID is not accepted as proof of randomness without the documented generation assumptions. OpenTelemetry's probability-sampling specification provides explicit mechanisms and qualifications; its development status and deployed SDK support still need checking. Do not promise an estimator merely because a UI shows a sampling percentage.

For an independent fixed-probability example, Reset the sampling lab, set both p=0.2 and n=25, predict retention-fraction mean 0.2 and variance 0.0064, then Play or Step. Reset and set both p=0.2 and n=100 to predict variance 0.0016. The exact binomial masses describe hypothetical independent decisions, not actual random draws; playback highlights outcomes. Its normal overlay is approximate, and the displayed Berry–Esseen bound concerns standardized CDF error, not per-bar error. The instrument does not model tail policies, correlated traces, adaptive rates, or inverse weights.