Static shapes and memory planning

For a contiguous tensor, multiply every dimension by the element width, then account for the required alignment and any backend-specific padding or layout. A maximum input shape can propagate into larger intermediate bounds. ExecuTorch 0.7’s memory-planning documentation describes shape specifications, bounded symbolic evaluation, and arena assignment. The bound must cover accepted runtime inputs. Merely tracing one small example does not justify allocating for that example while permitting larger inputs through the application interface.

Draw a timeline showing when each tensor is produced, last read, and released. Reuse requires nonoverlapping lifetimes or an explicitly legal in-place operation, not simply two equal sizes. A producer’s input and output often coexist while its kernel runs. If a delegate or DMA engine consumes a buffer asynchronously, the lifetime ends only after completion under the relevant synchronization contract. External callers retaining an output also extend its lifetime beyond the model’s internal schedule.

The pinned TFLM arena documentation distinguishes nonpersistent head allocations, scoped temporary allocations, and persistent tail allocations. Its planner reuses tensor storage across known lifetimes, and recording APIs help inspect allocation categories. This is not a claim that application stacks, sensor buffers, all backend allocations, or model constants always live in that arena. Keep a complete device map and identify each owner. Likewise, excluding ExecuTorch method inputs or outputs from a plan moves their storage obligation to the caller.

The maximum sum of simultaneously live tensor sizes is a useful lower bound, but alignment, contiguous-region requirements, fragmentation, scratch workspace, and planner heuristics can require more. Add persistent state and concurrent runtime/application allocations at their actual overlaps rather than blindly adding independent maxima from unrelated phases. Initialization may also need temporary conversion or repacking space. Measure high-water marks on the chosen device and test the largest supported shapes, multiple streams, and failure paths where lifetimes can lengthen.

Reject out-of-contract shapes before execution, or select another independently validated plan. Test repeated calls and interleaved sessions because a plan safe for one invocation can fail when two use the same arena concurrently. Use canaries or memory tools appropriate to the target, inspect planned offsets and lifetimes, and compare outputs under stress. MCUNetV2 illustrates how changing inference scheduling can reduce peak memory; that result does not make an arbitrary reused buffer safe or transfer its measured savings to every network.

The cache lab illustrates reuse under a fixed capacity, not lifetime allocation. Reset, set capacity C to 4 and stride s to 4, predict four misses and twelve hits in the initial sixteen requests, then Step: the seventeenth request is another hit. Compare a fresh Reset with both C at 2 and s at 4 before Step: all seventeen miss. This fully associative LRU toy has no tensor graph, arena offsets, scratch space, or asynchronous consumer ownership.