Memory, locality, and the shape of data

A dense rectangular array has a simple element count: multiply its dimensions. Multiply again by bytes per element to obtain its raw payload. This estimate excludes object headers, alignment, sparse indices, and runtime bookkeeping. In training, weights are only one category: activations, gradients, optimizer state, input batches, communication buffers, and temporary workspaces can all consume memory. Count their lifetimes as well as their maximum individual sizes.

Two objects with the same logical contents need not use the same layout. An array of packed numbers differs from a collection of references to separately allocated objects. A view can reuse existing storage while a copy allocates new storage. An apparently harmless conversion or transpose may create a copy in one library and a view in another. Check the actual operation's contract before assuming either behavior.

Virtual memory gives a process an address space whose mappings are managed by the operating system. Physical pages can back those mappings, and pages may also be shared or backed by files. Reserving a large address range is therefore different from actively occupying the same amount of physical RAM. Metrics with similar names can count different things; compare resident usage, virtual size, and allocation reports only after understanding their definitions.

Caches and the memory hierarchy exploit reuse and nearby access, but exact sizes and policies depend on hardware. Reading contiguous elements often gives the system more opportunities to use fetched data than following widely scattered references. This is a mechanism to investigate, not a promise that every contiguous program is faster. Arithmetic intensity, vectorization, prefetching, parallel access, and working-set size also change the outcome.

A working set is the data actively needed over a period of execution. Processing in blocks can reuse a smaller working set before moving on. Reusing a buffer can reduce repeated allocations, while careless reuse can overwrite data still needed elsewhere. Ownership and lifetime rules therefore belong alongside performance reasoning: know which component may mutate a buffer and when readers have finished with it.

Before changing layout, measure the actual bottleneck. A program waiting on a remote database will not necessarily benefit from a packed local array. The lab offers a related capacity example: a dense all-pairs table has a quadratic element count. Interpret positive $x$ as a normalized item count and $a$ as a normalized per-entry byte cost in $y=ax^2$. Move $x$ to observe how a doubled item count multiplies payload. It is not a cache or bandwidth simulator. Compare actual throughput and peak live memory separately when changing a real layout.