Choose a design point that meets the whole contract

Specify the task, evaluation data, quality metric, request shapes, arrival process, latency statistic, and required throughput. Include memory capacity, numerical tolerance, availability, and supported operators where they constrain deployment. A design that misses one mandatory requirement is infeasible for that decision even if it has excellent arithmetic efficiency. Keep its result in the research record, but do not rank it as a cheaper replacement for a system meeting the original contract.

Separate controlled ablations from complete-system comparisons. Holding the model fixed can isolate a mapping or hardware mechanism; changing quantization, architecture, or sparsity explores another design point. Both are valuable when labeled. Timeloop illustrates systematic architecture and mapping exploration, while MLPerf Inference defines concrete quality and scenario-based performance methodology. Neither eliminates the need to disclose the actual workload, versioned rules, tuning budget, and measurements used in a particular co-design claim.

For a steady interval, average included power divided by completed request rate yields energy per completed request. Include host, memory, networking, conversion, and idle allocation according to the declared boundary. If only successful deadline-qualified requests matter, use their rate and assign all spent energy to that same interval. Do not quietly divide one design’s board power by another design’s best kernel throughput. State whether replication, spare capacity, and utilization losses are part of the deployment budget.

Cost per request depends on provisioned capacity and achieved utilization as well as nominal device price. Include the hardware or rental boundary, software support, power, and any required auxiliary systems consistently. A design with lower cost per completed request at saturation may still need more replicas to handle a latency target or failure budget. Use the expected deployment demand rather than assuming every purchased device runs continuously at the benchmark’s maximum useful rate.

An analytical ceiling, cycle simulation, synthesis estimate, implemented prototype, and production workload measurement support different claims. Describe the method and uncertainty for each design point. Calibrate cost models on representative observations, reserve validation cases, and explain residual errors. A simulation can guide search without proving physical timing or power. A measured prototype can prove one implementation’s behavior without establishing a result for a larger, differently connected system.

Release model artifacts, compiler and driver versions, placement plans, numerical settings, dataset IDs, warm-up, measurement intervals, and raw per-run summaries. Repeat experiments enough to characterize relevant variation and inspect tail behavior at the intended load. Report the feasible Pareto set rather than declaring one universal winner across incompatible objectives. The final choice should follow the application contract and evidence, while unresolved assumptions remain visible for the next experiment to test.