Launch overhead and graphs: amortize a repeated dependency structure

A CPU can enqueue a kernel and return before the device finishes it. Timing that host call therefore measures submission behavior and any blocking, not the kernel’s full execution. For many small dependent kernels, gaps in submission may leave the device waiting; for a large compute-bound kernel, reducing host launch cost may barely affect application latency. Use completion-aware measurements and distinguish cold initialization, compilation or loading, warm repeated submission, and steady device execution.

A graph contains nodes and dependency edges for supported operations. Instantiation prepares an executable graph, and a graph launch submits that executable structure under its documented semantics. Independent nodes expose potential overlap while dependent nodes retain order. Capture has restrictions, and graph-update support depends on which parameters or structure change. Do not assume arbitrary dynamic control flow can be captured once and replayed unchanged forever. Check CUDA12.6 graph rules for the actual node types, allocations, and stream behavior in use.

Graph creation and instantiation add an up-front cost. If a workload replays the same valid graph many times, a lower per-iteration cost can amortize that setup. If shapes change frequently and require updates or reconstruction, those costs belong in the comparison. A graph may also reduce some device-side scheduling overhead, but that is measured behavior, not an invariant of the API. Keep numerical outputs, buffer addresses, lifetimes, and dependency edges correct when changing parameters across replay.

Kernel fusion is a different transformation. Combining stages can remove launches and intermediate traffic but change register pressure, occupancy, and synchronization opportunities. A graph preserves a supported operation structure rather than automatically fusing all kernels into one. Compare both approaches when relevant, but do not erase the costs one approach retains. For an application, include the final output completion and any CPU work that remains on the critical path instead of reporting only the smallest enqueue measurement.

Benchmark repeated warm iterations and the actual frequency of shape or parameter changes. Record whether each update is supported, whether instantiation repeats, and whether memory allocation or synchronization sneaks into the loop. Preserve an ordinary execution path for comparison and test error handling during graph construction and launch. A graph that reuses a freed input buffer is not an optimization regardless of its timing. Focus on useful completed iterations per second and relevant latency distribution.

The queue lab illustrates repeated submission pressure only. Reset, set λ=4 and μ=3, predict backlog 4+t, then Play or Step. Reset and set λ=2 and μ=3 for drainage. This fluid work model does not build or launch graphs, measure overhead, or fuse kernels. It shows why reducing unnecessary arrivals or per-item service work can matter, but the claimed graph benefit still needs the explicit setup-and-replay measurements in the worked example.