Synchronization scopes: order the right memory for the right participants

A block barrier such as __syncthreads coordinates required non-exited threads in that block and provides the documented visibility for their relevant earlier accesses. It is not a grid-wide barrier between arbitrary blocks. Calling it in a conditional reached by only part of a still-participating block can violate the contract. Warp synchronization similarly requires a valid mask and matching participation. A warp shuffle or vote with a sync suffix is not automatically the same memory-ordering primitive as a shared-memory barrier.

A memory fence orders a calling thread’s specified memory operations at its documented scope; it does not make all other threads arrive or wait for a producer to finish. Publishing data through a flag therefore needs a correct atomic communication protocol, not a plain store plus the word fence in a comment. Volatile accesses do not generally replace atomics or establish the required happens-before relationship. Choose a supported memory scope that includes both producer and consumer.

Threads within one block may use shared memory and a block barrier. Threads in different blocks require a supported device-wide communication protocol or a separate correctly ordered kernel phase; one block’s barrier cannot synchronize the other. Device-scope atomics do not automatically synchronize a CPU reader or a different device. System scope requires supported hardware, memory, and atomic semantics for all participating agents. Do not use a broad phrase such as globally visible without naming which observers and which operation establish the relation.

Host completion is another boundary. cudaStreamSynchronize waits for preceding work in one stream; cudaDeviceSynchronize waits according to its device/context scope and may report asynchronous errors. Neither repairs a data race that already occurred inside a kernel. A host thread also must not consume an asynchronous copy’s destination or reuse its source before the relevant completion. A synchronous-looking CPU call and a device memory fence are different mechanisms with different participants and costs.

For each shared value, identify its producer, consumers, storage location, and lifetime. Draw the ordering edge that makes the write visible before the read. Check every branch around barriers and every mask under partial warps. For inter-block cooperation, avoid a busy-wait scheme that assumes all blocks are simultaneously resident; waiting blocks can occupy the resources needed by the producer. Test with varied block sizes and inputs, then use diagnostic tools within their documented coverage. Fewer barriers are only an optimization if the required ordering remains true.

The network lab shows a mathematical dependency, not concurrent visibility. Reset, set x=1 and w=1, predict output sigmoid(0.5)≈0.622, then Step. The displayed output depends on its input computation, but the diagram has no parallel memory agents, barriers, atomics, or fences. Its limited role is to encourage an explicit dependency chain. The actual GPU proof must add the memory-order and scope edges that a simple function graph does not contain.