Reverse execution needs the forward values its rules depend on

For multiplication, input values help compute the reverse contributions; for an exponential, its output can provide the local derivative. Different implementations can save different sufficient residuals while representing the same derivative. An abstract tape records operations and the information needed to reverse them, but a compiled runtime can fuse, simplify, or rematerialize parts of that structure. Do not assume a one-to-one mapping between Python statements, graph nodes, kernels, and saved allocations. The mathematical dependency is stable while the representation and optimization policy vary.

PyTorch 2.9 documents saved tensors as forward intermediates needed by backward rules. It may pack and unpack tensor objects to avoid reference cycles, so object identity is not a reliable API for determining whether underlying storage is shared. A saved tensor can retain a large backing allocation even when its visible slice is small. Conversely, several logical saved views can share one allocation. Inspect storage and lifetime evidence when diagnosing memory rather than summing every printed tensor shape and treating that sum as exact resident bytes.

A graph can remain reachable because an application stores an output, loss, closure, or diagnostic object that references its history. Ordinary backward often releases saved resources when they are no longer needed, subject to graph-retention settings and framework behavior. Retaining a graph can keep memory live longer, but saving more graphs is not the same as recording a differentiable graph of the derivative. These become distinct choices for higher-order work. Avoid keeping unnecessary training outputs across iterations when only a detached scalar statistic is needed, while preserving the intended derivative dependencies.

A memory budget should separate parameters, gradients, optimizer state, saved activations, temporary workspaces, allocator reserves, and application references. The total peak depends on overlapping lifetimes, not the sum of each category's individual peak measured at different times. Saved-tensor hooks and offloading can alter where residuals live and what reconstruction costs later; they must preserve the values required by backward. A lower accelerator-memory number can come with host-memory or transfer costs. Measure both derivative correctness and peak resource usage under the actual execution schedule.

For a small trace, write which forward value each reverse rule uses and when that value can be discarded. Compare a repeated backward request, a retained graph, and a deliberately recomputed forward evaluation under the framework's contract. A value that prints correctly now may already have lost the historical value needed for backward after mutation. An error about a freed or modified saved tensor is evidence about a specific lifetime dependency, not a suggestion to turn retention on everywhere. Identify the intended schedule before choosing a fix.

The network lab gives one inspectable sigmoid, not a reverse runtime. Reset, set BOTH input x=1 and weight w=2, predict z=wx−0.5=1.5 and output about 0.8176, then Play or Step. On paper, its input derivative is wσ(z)(1−σ(z))≈0.2983. Reset and re-set BOTH x=0.5 and w=2 to inspect a different forward value. Edits retain t; playback traces the fixed graph without computing a backward tape or saving training activations. Explain which selected primal values a real local rule would require rather than treating the animation as memory evidence.