Place a workload graph rather than a peak operation count

Represent each operation or fused region as a vertex and each produced tensor as a directed edge. A placement assigns vertices to hardware resources; a crossing edge adds data movement, conversion, and synchronization as needed. The same-device case may avoid an interconnect transfer while still paying local memory traffic already included in its execution estimate. Keep units and data representations explicit: an activation in FP32 has different byte volume from its quantized counterpart, and layout conversion can require another kernel.

Estimate operation costs at the actual tensor shape, precision, batch, and compiler configuration. Peak arithmetic throughput is only a ceiling for a compatible operation mix. Small kernels may be dominated by launch costs; irregular kernels may fail to expose enough parallelism. Profile representative regions or use a calibrated model with uncertainty. An operation placement that looks fastest in isolation can also increase its neighbors’ transfer volumes, so optimize graph boundaries rather than selecting the locally fastest device independently for each vertex.

A mapping must fit weights, live activations, scratch buffers, and transfers in each memory tier. Tensor lifetimes matter because producer and consumer overlap can keep multiple versions resident at once. A placement that exceeds capacity may require tiling, spilling, or a different batch size, each changing execution and transfer costs. Account for replicated constants and temporary conversion buffers rather than checking only the final model-file size against advertised memory capacity.

Two independent vertices cannot necessarily run simultaneously merely because the DAG permits it. They may share a GPU, a CPU worker, a memory controller, or one link. Model those resources and produce an actual schedule, or label the result as a lower bound. Pipeline overlap can improve throughput while leaving single-request latency larger. State which metric is being optimized and include any admission queue when the target is a loaded service rather than an isolated request.

Search can use measured candidates, dynamic programming for restricted graphs, or broader optimization. Systems such as Alpa demonstrate that inter-operator boundaries and intra-operator partitioning interact; their design is evidence for structured search, not a guarantee that one cost model predicts every heterogeneous machine. Prune infeasible mappings early, and retain the assumed costs with each chosen plan. A winning plan is difficult to debug if the compiler discards the explanation for why its crossing edges were expected to be cheap.

Validate end-to-end latency, sustained throughput, memory high-water marks, and output quality on the final mapping. Compare against an optimized baseline using the same request population and precision. When an estimate misses, identify whether a transfer, launch, conversion, or contention term changed instead of multiplying the whole model by a convenient correction factor. Report sensitivity to batch and working-set size so a plan selected for a large batch is not presented as universally best for interactive requests.