Device-cache reuse and bypass

A working set is the data needed over a useful execution interval. A tile reused many times may benefit from cache residency, while a streaming pass much larger than the cache can evict reusable data. Access order and concurrent blocks change the effective reuse distance. Record whether the same bytes are revisited before enough competing data arrives to displace them. A cache hit rate aggregated over a whole kernel can hide an important tensor that misses frequently or a high-rate stream with little performance impact.

Coalescing and reuse interact but remain different. Adjacent warps may reuse sectors over-fetched by a misaligned access, reducing external-memory traffic relative to a naive per-warp sum. Shared memory can provide explicit block-local reuse when the algorithm justifies its synchronization and capacity cost. Do not assume an L1 hit removes all latency or that an L2 hit has the same cost as a register access. Measure traffic and stalls at the level relevant to the optimization question.

PTX load cache operators express documented cache behavior or hints, with support and implementation depending on the target. For example, load operator .ca requests caching at all levels, while .cg requests caching at the global level rather than L1; the documented distinction still retains L2. These PTX 8.7 operators are performance hints and do not change memory-consistency semantics. Do not describe a global-level cache operator as bypassing every cache or forcing every access to DRAM. Store policies and specialized access paths have separate semantics, so transferring one load rule to every instruction is unsafe.

Cache policy does not establish producer-consumer synchronization. A reader can still race with a writer even when a cache is bypassed, and a volatile qualifier is not a substitute for the required atomic ordering or synchronization scope. A performance hint cannot make stale or out-of-lifetime accesses correct. Check the memory model and completion dependencies first, then compare cache-policy variants on equivalent valid executions. Also record how cache configuration trades resources with shared memory on architectures where those capacities interact.

Warm and cold runs answer different questions. Control initialization, preceding kernels, input size, and measurement synchronization so that a claimed cache improvement is not simply a different starting state. Test whether the optimization survives other representative shapes and concurrent work. A policy that favors one persistent tensor can harm another. Cache-bypass experiments should therefore report the workload’s reuse hypothesis and measured consequences, rather than a universal rule to disable or enable a cache for all AI kernels.

The cache lab is an exact fully associative LRU toy, starting empty with sixteen requests already counted. Reset, set capacity to 4 and stride to 4, predict hits after seventeen requests, then Step once. Compare a fresh Reset with both capacity at 2 and stride at 4 before another Step. Each Step adds one request here. The four-address cycle fits the first cache but not the second. This instrument is not a GPU cache: it has no sectors, associativity conflicts, cache operators, or hardware replacement policy.