Compare agreement gains with the full resident resource bill

Perplexity, parameter count, and top-token agreement can help characterize a draft but do not directly determine the exact proposal overlap sum min(p, q). Evaluate that overlap or observed verified-prefix behavior on representative prompts, temperatures, constraints, and context lengths. A draft trained for one domain may be cheap but poorly matched elsewhere. A more accurate language model can still have an unfavorable runtime ratio on the chosen hardware. Keep target and draft tokenization compatible with the selected verifier, and measure any supported translation or mapping cost separately.

Proposal mechanisms include separate smaller models, trained auxiliary heads, early-exit paths, and prompt-based lookup under particular supported algorithms. They do not all have the same probability information or correctness argument. A shared backbone can avoid duplicating some weights, while adding heads or jointly fine-tuning a backbone can alter what baseline model is being preserved. Distinguish an unchanged target plus verified proposals from an updated target whose quality is assessed against an older release. Pinned API support and paper results apply to their stated models and implementations; do not infer that every assistant choice has the same compatibility or performance.

For a conventional decoder with keys and values stored per layer, cache bytes scale with layer count, KV-head count, head width, scalar size, and retained tokens across requests. Use KV heads rather than query heads for grouped-query or multi-query attention. Compression, quantization scales, sliding windows, and architecture-specific state can change the estimate. The draft has its own cache unless a supported implementation shares state correctly. Temporary verification tokens and branches also need storage even when later rejected, so peak memory can exceed the committed-prefix cache size.

Logits can be substantial when many candidate positions each produce a vocabulary-sized vector, particularly if sampling calculations use a wider dtype than stored weights. Workspace, graph captures, communication buffers, and fragmentation also affect the feasible batch. A memory budget should state which allocations coexist and what remains available for request growth. Reducing weight precision can change both memory and proposal probabilities, so remeasure overlap and numerical behavior instead of assuming a storage reduction preserves every previous verification property. The target model and its declared law remain the reference for exactness.

Compare draft alternatives using committed tokens per time, peak memory, time to first token, inter-token latency distribution, and throughput at the intended load. A proposal that saves target weight reads for one isolated request may interfere with a saturated batch. Include low-acceptance and short-output requests where overhead dominates. Record model and backend versions and keep a non-speculative fallback under the same target settings. A higher acceptance percentage is one diagnostic, not a release decision detached from the added execution and capacity costs.

The roofline lab illustrates the resource distinction only. For each separate view, Reset, set both bandwidth B=100 GB/s and operational intensity I to 2 or 10 FLOP/byte, predict, then inspect. The synthetic ceilings are 200 and 1000 GFLOP/s with a fixed 1000 GFLOP/s compute roof. This model has no draft, acceptance process, cache allocation, or measured accelerator. It shows why additional useful arithmetic may fit a bandwidth-bound regime yet compete with a compute ceiling; actual draft selection still requires the full memory and latency measurements just described.