Publish an energy result another experiment can challenge

Record code and model artifacts, tokenizer and data versions, useful-output counts, precision, batch policy, cache state, concurrency, and quality checks. Hold service constraints and the completed work definition fixed, or report intentional changes explicitly. Warmup and compilation belong either inside the declared measured service or in a separately documented amortization. An identical shell command can still execute a different workload when remote data, model versions, or runtime defaults change.

Record meter placement or API scope, device identity, driver and interface versions, counter epoch, timestamps, integration method, and start/stop synchronization. Confirm that asynchronous work has completed before closing the intended window. Preserve invalid reads and counter resets as diagnostic events rather than replacing them with zero. For facility or emissions accounts, store allocation policies and immutable factor versions with their units, geography, time resolution, and gas scope. This separates reproducible arithmetic from unsupported input assumptions.

Pair runs under comparable conditions and randomize or counterbalance their order when appropriate. Thermal drift, background activity, and time-of-day effects can otherwise become confounded with implementation. Independence across pairs is an assumption to examine; rapid repeats with shared cache state or common external load can be correlated. A small standard error from many dependent runs is not evidence that the absolute energy measurement is accurate.

NIST’s uncertainty guidance distinguishes evaluations based on statistical observations from those based on other information such as calibration. A persistent meter bias does not generally shrink as the number of repeated runs grows. A common additive bias can cancel in a paired difference if it truly applies equally, while a differential bias can remain. State which uncertainty sources are shared and which are not instead of claiming either universal cancellation or universal independence.

Publish gross totals, useful-work denominators, run-level observations, and rejected-run reasons before compressing the study into one ratio. Report uncertainty sources separately when combining them would require unjustified distributional assumptions. A narrow repeatability estimate can coexist with a wide factor or calibration range. Rounding a final number appropriately is more honest than retaining every decimal from multiplication when the dominant input came from a coarse assumed factor.

A robust comparison also explains its scope of generalization. One matched input mix, device generation, facility period, and driver version do not establish a universal ranking. Reproduce the relevant result on other representative workloads when making broader claims, and record failures as well as wins. Preserve configuration and data snapshots so the experiment can be rerun after an implementation change. The objective is an auditable decision with stated limits, not an unchallengeable environmental label.