Edge device resource and latency profiles

Flash or persistent storage holds model files and application code; RAM holds active tensors, interpreter metadata, stacks, input buffers, and kernel workspace. Some platforms access constants directly from mapped storage, while others copy or transform them into RAM or accelerator memory. Do not count a compressed model file as the working-memory requirement. Measure initialization and steady execution separately because unpacking, compilation, or weight repacking can create a larger startup peak than a single warm inference.

Include the rest of the application in the budget. Camera buffers, audio capture, radio activity, UI rendering, and operating-system services compete with inference for memory, bandwidth, and scheduling time. A microcontroller without virtual memory and a mobile application managed by an operating system need different failure handling. Report what was reserved and what was merely observed free during one run. A successful allocation in a quiet laboratory is not a guarantee that the deployed workload always has the same resources.

Compute work and transferred bytes provide useful feasibility checks. Under a declared arithmetic convention, divide operations by a compute ceiling and bytes by an applicable bandwidth ceiling. With ideal overlap, the larger duration is a lower bound, not the sum; dependencies, cache misses, launch overhead, and contention can increase actual latency. Specify whether a multiply-add counts as one or two operations. Comparing a model’s MAC count with a device’s FLOP rate without that convention creates a factor-of-two error before measurement begins.

Operational intensity relates arithmetic work to bytes moved at a specified memory boundary. It can differ between external memory, shared cache, and accelerator-local storage. A nominal bandwidth and a peak arithmetic rate need not be simultaneously attainable for the chosen kernel. Profile representative inputs and tensor layouts, including preprocessing and transfers. Keep cold start, first invocation, warm isolated inference, and continuous operation distinct; their bottlenecks and memory footprints can differ enough to require separate acceptance criteria.

State arrival rate, batch size, input shapes, concurrency, quality targets, and deadline policy alongside the device settings. Record battery state, ambient temperature, enclosure, and whether charging or radios are active. Measure latency distributions and missed deadlines over a useful duration rather than reporting only the fastest run. The MLPerf Tiny and Mobile papers motivate joint quality and performance evaluation, but a product’s sensor pipeline and duty cycle can require additional measurements beyond a benchmark’s defined workload.

The roofline instrument is a synthetic ceiling illustration. Reset, set intensity I to 2 and bandwidth B to 80, predict performance ceiling 160 GFLOP/s, then Step. Compare a fresh Reset with both I at 8 and B at 80 before Step: 640 GFLOP/s. Its compute roof is fixed at 1,000 GFLOP/s. These numbers do not describe an edge processor or estimate application latency; they show how a declared boundary and arithmetic intensity affect one idealized bound.