Model conversion and operator coverage
Validate exported semantics and backend assignments, accounting for fallback and partition-boundary costs rather than counting supported operators alone.
On this page
The question for today
What you’ll understand
- Separate export success from kernel and backend compatibility.
- Compute a mixed accelerated-and-fallback latency including boundary costs.
- Design output comparisons and unsupported-operator failure tests for a pinned runtime.
Exporting a model changes its representation and sometimes the operators used to express it. A successful conversion proves that a file was produced, not that every operation runs on the intended accelerator or preserves the application’s behavior. Inspect the converted graph, its input contract, and the actual runtime assignment. This lesson uses ExecuTorch 0.7 and ONNX Runtime 1.22.0 as named examples; their export and fallback mechanisms should not be assumed to describe every edge runtime.
Compatibility includes more than an operator name
An operator can be supported only for particular versions, data types, layouts, shapes, attributes, or quantization schemes. A convolution name in a support table does not imply support for every dilation, channel arrangement, or per-axis scale. Also verify preprocessing and postprocessing: channel order, image resizing, normalization, tokenization, and output interpretation can change quality without an operator failing. Preserve a small versioned input-and-output corpus that exercises both normal and boundary cases through the complete exported pipeline.
Compare the source model with the converted reference implementation before introducing accelerated kernels. Then compare the selected device backend, using justified numerical tolerances and task-level metrics. Exact equality may be inappropriate across floating-point implementations, while loose tolerances can hide a layout or scaling error. Record exporter version, model opset or dialect, runtime build, delegate configuration, and kernel selection. If a graph rewrite removes or fuses operations, verify the rewritten graph’s semantics rather than relying only on the original module list.
Fallback is an explicit deployment behavior
ONNX Runtime partitions graphs according to execution-provider capability and provider order; a CPU provider can execute supported operations left outside an accelerator partition. That is graph assignment, distinct from the Python session’s runtime execution-provider failure fallback. Pin which mechanism is enabled and inspect the session’s actual behavior. A restricted build or a missing CPU kernel can still fail. Do not interpret the presence of an accelerator provider as evidence that all nodes execute there or that every failure has a working fallback.
ExecuTorch 0.7 exports and lowers a captured graph, with backend-specific delegation and portable kernels for the applicable remainder. Its documented restrictions on dynamic control flow and backend state mutation are part of the contract. TFLM’s selected operator resolver is another distinct case: an unavailable required kernel is not rescued by a universal desktop interpreter hidden underneath. Treat unsupported cases as tested initialization or execution failures, or ship a deliberately chosen alternative model whose quality and resource costs have been validated.
Inspect the deployed path under realistic inputs
Report time-weighted assignment and boundary overhead along with the supported-operator count. One unsupported operation can dominate runtime or split an otherwise efficient accelerator subgraph into several pieces. Profile cold compilation and warm execution separately, and test shapes that may select different kernels or partitions. A fallback that preserves numerical outputs can still violate the product’s deadline, power, or memory budget. Release criteria should therefore include both correctness and the actual device execution path.
The queue lab illustrates only the capacity consequence of a slower service path. Reset, set arrival lambda to 4 and service mu to 3, predict backlog five after two Steps, then take those Steps. Compare a fresh Reset with both lambda at 2 and mu at 3 before two Steps: backlog three. Initial backlog is four and two Steps are one second. No operators are converted or partitioned here; service rates are supplied hypothetical flow parameters, not measured fallback performance.
Work it through
High operator coverage still pays the slow remainder
A baseline inference takes 100 ms. Operations occupying 80 ms are accelerated eightfold; the remaining work takes 20 ms. Two sequential partition boundaries each add 2.5 ms for copies and synchronization. Assume no overlap and unchanged numerical quality for this timing exercise.
The accelerated work takes 80/8 = 10 ms. The unchanged remainder still takes 20 ms, regardless of its operator count.
Boundary cost is 2 × 2.5 = 5 ms, so mixed execution takes 10 + 20 + 5 = 35 ms.
End-to-end speedup is 100/35 ≈ 2.857, matching f = 0.8, S = 8, and beta = 0.05. It is smaller than the isolated accelerated portion’s eightfold speedup.
Coverage must be interpreted through executed work and boundary costs rather than a percentage of graph nodes.
The living notebook
Explore the model
Interactive lab · queue
A supplied fallback rate can create backlog
Reset, set λ = 4 and μ = 3, predict Q(1) = 5, then take two Steps. Compare a fresh Reset with both λ = 2 and μ = 3 before two Steps: Q(1) = 3. Initial backlog is 4 and each Step is 0.5 seconds. This fluid queue does not export models, partition operators, or measure accelerator fallback.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forNumerical fallback correctness and sufficient service capacity require separate evidence.
Pause & reason
Check your understanding
Your turn
Try it yourself
A 60 ms baseline spends 45 ms in work accelerated fivefold and 15 ms in an unchanged remainder. Boundary work adds 6 ms. Compute final latency and speedup, then name one correctness check unrelated to speed.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Add accelerated time, unchanged time and new boundary cost under the sequential assumption.
Compare with the solution
Accelerated time is 45/5 = 9 ms. Final latency is 9 + 15 + 6 = 30 ms, giving speedup 60/30 = 2. Compare source, converted reference, and device outputs on versioned boundary inputs with justified tolerances, and verify preprocessing, class order or other task semantics. A twofold speedup cannot compensate for a converted model receiving the wrong tensor layout or input normalization.
Keep the idea
Take it with you
- Check operator versions, shapes, types and attributes against the exact backend.
- Distinguish graph assignment from runtime failure fallback.
- Measure the complete converted path, including its remainder and boundaries.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.