Prefetching, timeliness, and wasted work

Hardware prefetchers infer useful patterns from observed accesses, using implementation-specific logic. A sequential stream is often easier to anticipate than a chain whose next address is stored in the current node. Software prefetch instructions give another possible hint, but their semantics and usefulness depend on the target architecture. They generally do not replace correct address ownership or synchronization. A hint must not become an excuse to access objects outside their valid lifetime or ignore the language's rules.

Accuracy asks what fraction of issued prefetches later prove useful under a declared accounting rule. Coverage asks how many relevant misses the mechanism avoids relative to a baseline. Timeliness asks whether requested data arrives before demand without being evicted too soon. These dimensions differ: a perfectly accurate request issued after the demand cannot hide its latency, and an extremely early request may displace something more valuable. Define the denominator before comparing reported prefetch percentages.

A streaming kernel may overlap several outstanding fills while arithmetic continues on earlier data. Its achievable overlap is limited by dependency chains, request-tracking entries, cache capacity, and memory bandwidth. If an address cannot be computed until a previous miss returns, simply asking for more simultaneous requests does not create independence. Reorganizing data or processing several independent chains can change the dependency structure, but may add bookkeeping and alter the working set.

Useless prefetches still compete for some combination of link bandwidth, fill buffers, and cache space. A workload already saturating memory bandwidth may gain little from fetching more aggressively. A useful benchmark therefore reports elapsed time and relevant traffic, not just fewer demand misses. Demand misses can decrease because prefetching brought lines early while total bytes increased enough to slow the program. Consider co-running workloads as well as an otherwise idle machine.

First establish a representative access pattern and an observable result that prevents dead-code elimination. Compare warm and cold conditions deliberately, and record working-set size and placement. If supported controls let you change a prefetch mechanism, document exactly which one changed and repeat the measurement; a global switch may affect multiple streams. Large changes in instruction count, layout, or thread scheduling make a simple before-and-after claim harder to interpret.

The queue lab models fluid backlog from an initial four jobs using arrival rate λ and service rate μ. Reset, set λ=4 and μ=2, predict growth, then Play or Step. For a separate comparison, Reset again, re-set both rates to λ=2 and μ=4, predict the decline from four, then Play or Step. Control edits recompute the whole trajectory at the current elapsed time; they do not switch rates within a retained backlog history. Use this only for the abstract lesson that submitting work faster than service capacity creates pressure. It has no cache, request buffers, address prediction, or dependent instructions. Its separate M/M/1 readout adds stochastic assumptions and is not a CPU miss-latency formula.