A profiler changes execution, so measure the observer's cost

Measure a warmed baseline and a comparable instrumented window with the same model work, data shapes, update count, and completion boundary. Report the ratio rather than assuming a tool's advertised overhead applies to every workload. A fixed per-event collection cost can be substantial for many short kernels and less visible for long operations. Collection can also change scheduling, induce waits, or add memory pressure, so subtracting one average overhead number from every event does not generally reconstruct an uninstrumented timeline.

Distinguish startup from recurring cost. Profiler initialization, metric configuration, trace export, and warmup can occur outside a chosen active window while still affecting a full-run benchmark. Record the scope of each overhead estimate. If the instrumented run changes batch size or drops work to fit its buffers, it is not a matched comparison. Keep an ordinary completed-work measurement as the authority for final throughput, and use the profile to explain mechanisms with its intervention clearly documented.

Framework shape and stack collection can add host work, and retaining references for shape recording can change tensor lifetime or optimizations. Kernel metric tools may replay launches or serialize execution to collect counters unavailable together. The profiled wall interval can then include save-and-restore work and repeated passes that ordinary execution never performs. Different replay modes have different concurrency semantics. Consult the deployed tool's versioned contract before comparing its wall clock with a timeline collector or assuming that a measured kernel duration includes the same boundary.

Sampling reduces how much is observed but can miss short or rare events, and periodic samples can alias a periodic workload. A simple random step-sampling model gives a useful expectation only if selection is independent of the event and inclusion probability is known. Capturing only visibly slow steps intentionally selects a conditional population; it cannot estimate the ordinary step distribution without an appropriate design. Buffer loss, unsupported activities, and capture windows create further missingness. Absence from a profile is therefore different from proof that an event never occurred.

Use a short system-level timeline to identify whether readiness, host launch, device execution, communication, or memory deserves deeper study. Then select the ranks, operations, counters, and capture range needed for that hypothesis. Compare the low-detail and detailed execution paths for changed overlap or lifetimes. Export outside a critical measurement window when appropriate, but report full-run export cost if operational monitoring overhead is the question. Preserve collection settings and version details so another engineer can understand both what was visible and what the observer changed.

The queue lab illustrates one possible buffering pressure, not a profiler implementation. Reset, set both arrival λ=4 and service μ=2, predict initial backlog four and backlog five after one Step, then inspect. Reset and set both λ=1 and μ=3 for backlog three after one Step and zero after two seconds. The fluid formula is max(0,4+(λ−μ)t), with no finite buffer, drop policy, trace export, or replay. Its separate stable M/M/1 residence readout uses additional stochastic assumptions. The model can illustrate a collection pipeline falling behind, but cannot quantify actual profiler overhead or dropped records.