Asynchronous errors: find the failing work, not just the reporting call

Check every relevant CUDA API return value. After a kernel launch, inspect the documented last-error state with an appropriate call, understanding that cudaGetLastError clears that state while cudaPeekAtLastError does not. These calls do not wait for the launched kernel to finish, and an error they observe can also reflect earlier asynchronous work. A clean launch-side check is therefore not proof that the device executed every access correctly. Start from a known diagnostic boundary and avoid ignoring a prior error that contaminates the interpretation.

For diagnosis, synchronize the relevant stream or completion event immediately after the suspected work and check that result. This narrows the interval in which an execution failure could have arisen, but earlier outstanding work or context-wide consequences still require care. Broadly serializing execution can change a race’s schedule, so a bug disappearing under debug synchronization is useful evidence, not proof of correctness. Preserve enough context to compare the original and instrumented schedules.

Compute Sanitizer memcheck can identify invalid or misaligned accesses and related memory errors within its documented coverage. Racecheck targets supported shared-memory access hazards rather than proving all global-memory protocols correct. Initcheck targets supported uninitialized device-memory reads, and synccheck targets misuse of supported synchronization primitives. Run memory checks before interpreting synchronization reports when appropriate, because earlier memory corruption can obscure later behavior. Include line information and preserve the exact kernel configuration and input needed to reproduce the issue.

A clean tool run covers the exercised execution and the tool’s supported checks; it is not a proof about every input, architecture, or schedule. Some asynchronous faults leave a context unusable for continuing work, so clearing a last-error variable is not a repair for corrupted device execution. Follow the documented recovery guidance for the error, often restarting the affected process or context as required. Keep the first causal diagnostic evidence rather than flooding logs with secondary failures from later operations.

Reproduce with the same seed, dimensions, data types, stream mode, and allocation lifetimes. Minimize the input and kernel sequence while preserving the failure. Check tails, strides, pointer arithmetic, dynamic shared-memory size, barrier participation, and cross-stream reuse. Put a completion check after kernel A and then after B to narrow the first failing interval, while recording that the instrumentation changes scheduling. A late memcpy reporting an error does not justify blaming that copy if an earlier kernel already wrote outside its buffer.

The sampling lab illustrates only how a stated observation model changes with trial count. Reset, set p=0.5 and n=20, predict mean proportion 0.5 and variance 0.0125, then Step. Playback highlights exact binomial outcomes rather than running random GPU trials. The lab has no sanitizer, fault timing, or race scheduling, and real failures need not be independent Bernoulli events. Use it as a warning that repeated clean runs need explicit coverage assumptions, then rely on concrete bounds and diagnostic traces for this kernel.