Measure the system and the cost of observing it

Suppose a worker reports a long request latency. “Trace everything” is not yet a hypothesis. Ask instead whether the worker spent the interval runnable but off CPU, blocked waiting for a resource, or running expensive kernel work. Each possibility needs different evidence and boundaries. A scheduler wakeup and later scheduling-in event can help examine runnable delay; a request marker ties that interval to the application’s own operation. Choose the relevant task identity, CPUs, event classes, and time window before collecting. Record the kernel configuration and enabled tracer because event availability and field meanings are interfaces to inspect, not assumptions to import from a different machine.

Tracepoints provide named event records; function tracing follows selected function execution, and sampling observes selected moments or counted events. These methods answer different questions. A sample-heavy function may consume much CPU time without being the cause of one rare stalled request. Conversely, an event sequence can establish that a wakeup preceded execution while saying little about work outside the captured interval. Apply filters and bounded capture duration so evidence remains interpretable and cost stays proportional to the question. Check that filters actually installed as intended, especially when setting subsystem-wide filters whose fields may not exist on every event. Retain collection settings alongside the trace so another person can reproduce the experiment.

Kernel tracing commonly writes into finite buffers. A producer can outpace a reader, and an overwrite mode may discard old records. A trace with missing events can resemble a logically impossible lifecycle or hide exactly the burst you wanted to study. Inspect the appropriate overrun and lost-record indicators, buffer mode, and capture boundaries before treating an absent event as proof of absence. Enlarging a buffer can buy time but does not remove sustained overload; reducing enabled events or applying effective filters attacks the production rate. Lost-event metadata and collection interruptions belong in the result, even if the remaining trace looks persuasive.

Clock choice also affects interpretation across CPUs. A fast local trace clock may not provide comparable timestamps across all CPUs on every system. Select a clock suitable for the ordering claim, understand its cost, and avoid subtracting raw timestamps from unrelated domains. Align application markers, trace intervals, and aggregate metrics to the same measurement window. If a throughput counter covers sixty seconds but a trace covers only the final two, their averages cannot be directly combined as if both described the same workload. Warmup, retries, background jobs, and migrated tasks can change the population. Preserve those distinctions rather than smoothing them away with a single average.

Estimate first-order recording cost as event rate multiplied by average per-event CPU cost. The units cancel into CPU-seconds per wall-second, which can be expressed as a fraction of one CPU. This budget includes only the cost represented by that measurement: stack collection, symbolization, data copying, reader wakeups, storage, and cache effects can contribute elsewhere. Compare equivalent workload runs with tracing off and on, repeat them, and inspect latency as well as throughput. A negligible average CPU increase does not prove that rare interrupt-sensitive paths are unaffected. Conversely, a measured slowdown is not evidence that every individual traced function is intrinsically expensive.

Sampling reduces volume but introduces detection uncertainty. Under a deliberately simplified model where each of n opportunities independently retains a relevant event with constant probability p, the probability of no retained observations is (1−p) raised to n. Real time-based sampling can alias periodic activity, and observations often correlate, so the model is a reasoning aid rather than an automatic confidence certificate. Finish with a conclusion matched to the evidence: the selected worker experienced a measured runnable interval in the captured window, or a sampled run suggests a hotspot worth a targeted follow-up. Preserve the capture settings and missing-event caveats. A small, reproducible experiment usually supports a stronger engineering decision than a massive trace with unexamined collection failure.