Edge fleet quality and performance validation

Define the deployment mix across device revisions, operating systems, accelerators, runtime versions, input shapes, languages or sensor conditions, and workload duty cycles. Sample independent sessions or users where records are correlated. Oversampling a rare but costly device is useful for diagnosis, but its raw test-set frequency need not equal its deployment weight. Record those weights and their uncertainty instead of treating a convenient balanced lab matrix as a random fleet sample.

Use versioned quality labels and the actual captured input pipeline. Conversion, integer kernels, resizing, streaming resets and local fallbacks can produce different errors on different devices. Compare with the reference and prior release on matched cases where possible, while guarding against repeated tuning to a visible final test set. Report slice support and unresolved label ambiguity. A weighted average improvement does not authorize hiding a severe regression in a small but important group.

Include cold initialization, warm isolated inference and sustained load, with deadlines and arrival models appropriate to the product. Inspect which backend actually ran and whether fallback, transfer, thermal management or memory pressure changed the path. Keep latency percentiles and missed-deadline fractions separate from throughput. Do not pool measurements across different input distributions and then attribute a changed tail exclusively to the model. Repeated trials should preserve operating conditions or record the differences needed to interpret them.

State whether energy includes the whole device, a power rail, or only an accelerator estimate. Integrate measured power over the specified interval, with synchronization and idle baseline handling documented. If an incremental energy result subtracts an idle trace, record that baseline and its measurement uncertainty rather than silently labeling the result total energy. The original MLPerf Tiny and Mobile papers provide concrete joint quality-and-performance benchmark designs; their workloads do not replace the product’s own sensor, connectivity and safety-relevant tests.

Create gates for quality, supported shapes, memory peaks, sustained deadline misses, and update/recovery behavior on the relevant strata. Exercise offline operation, corrupted artifacts, unsupported operators, queue overflow and interrupted promotion. Archive model and preprocessor hashes, runtime and compiler versions, backend assignment, test inputs, power traces and device settings. Roll out in cohorts with a policy for stopping and recovering within the security floor. Field monitoring should reveal errors without violating the data policy established for the product.

The sampling instrument illustrates uncertainty in a supplied independent correctness rate. Reset, set p to 0.9 and n to 20, predict sample-mean variance 0.0045, then Step. Compare a fresh Reset with both p at 0.9 and n at 100 before Step: variance 0.0009. These are exact Bernoulli sampling distributions, not device tests or random draws. They contain no fleet weighting, correlated sessions, latency quantiles, power traces, or evidence that a real classifier achieves the supplied rate.