Check what a power sensor observes and how often it changes

NVIDIA’s NVML R535 reference documents device power in milliwatts and distinguishes architecture-dependent behavior. It describes a one-second average on Ampere other than GA100 and on newer GPUs, while GA100 and older GPUs use the documented instantaneous path. These statements are scoped to that interface version and supported hardware, not a promise about every telemetry tool. The reading covers the GPU and associated circuitry such as memory; it is not whole-node or automatic per-process energy.

The same reference provides a cumulative energy query in millijoules since the last driver reload, with full support on Volta and newer supported devices. Check the API result and device identity instead of treating unsupported output as zero. Counter differences require a continuous counter epoch: a reload, reset, wrap, or device change can invalidate subtraction. A cumulative reading may help integrate activity, but its resolution, accuracy, and coverage still require characterization for the actual device and driver.

A sampled trace replaces an unknown continuous curve with observations and an interpolation rule. The trapezoidal rule assumes a straight segment between adjacent observations for the estimate. It is exact for a genuinely piecewise-linear curve at those breakpoints, but no such guarantee follows from printing timestamps. A narrow burst can fit between every pair of observations. Repeating the same phases may miss the same burst on every run, producing a systematic error rather than independent noise.

Lang and Rünger’s original profiling research illustrates why short GPU functions and low-resolution measurement require explicit reconstruction assumptions. Its historical hardware timing is not a current universal NVML specification. For a new experiment, characterize update cadence and phase sensitivity on the actual setup. Oversampling a one-second average creates correlated repeated averages; it does not recover the lost subsecond waveform. Preserve raw observations and describe any smoothing or interpolation rather than presenting a smooth chart as additional information.

Synchronize the measurement boundary with actual completion of asynchronous device work. A host launch timestamp and a host return timestamp may bracket only submission overhead. A device event can measure a particular stream interval but does not by itself include CPU preparation or other concurrent work. The synchronization used for measurement may also perturb overlap and execution time, so record it. Warmup and tail activity need a declared treatment, especially when the sensor averages across adjacent windows.

Compare a sufficiently long, repeated workload against a separately instrumented reference when practical, and report how repetition changes cache or thermal conditions. Distinguish calibration uncertainty from integration error and background interference. Do not reuse the R535 accuracy statement for specific older GPU families as a blanket error bound for all modern devices. A well-designed report can say what its instrument cannot resolve, while still providing a valid longer-window energy comparison within the documented boundary.