advanced27 min studyLesson 515 in Engineering

Preserve reads of shared storage across tensor updates

Analyze two-dimensional views, read-after-write dependencies, functionalization, and safe buffer reuse using a concrete mutation counterexample.

On this page

What you’ll understand

  • Identify which matrix views overlap an updated element.
  • Explain when a read cannot move across a mutation.
  • Distinguish functionalized tensor values from the externally observable mutation contract.

An alias is another route to the same storage, not merely another name with equal numerical contents. A tensor view can share selected bytes with its base while exposing a different shape or stride pattern. Mutating the base can therefore change a later view read even when the view variable itself was never assigned again. Compiler analysis needs the access sets and ordering that make this relationship observable. Assuming every new graph value is independent storage is especially dangerous when an optimization moves reads or reuses buffers.

Describe the shared locations before changing the schedule

Let X=[[1,2],[3,4]], and let v be its first-column view rather than a copy. The view reads locations [0,0] and [1,0]. A write adding ten to X[0,0] overlaps one of those locations, so a later sum of v must observe the update under ordinary sequential semantics. A view of only the second column would not overlap that particular write. This distinction is about accessed locations and operation semantics, not whether two variables have different spelling.

Now compare a read before the mutation with a read after it. Their shapes are identical, but their expected sums differ. Moving the later read earlier is not common-subexpression elimination because the intervening write changes its inputs through shared storage. Conversely, a proof that a write touches a disjoint region can enable motion that a conservative whole-buffer alias model would refuse. Unknown aliasing generally requires conservative handling; inventing disjointness to remove a copy trades correctness for an unproved assumption.

R(v)={(0,0),(1,0)},W={(0,0)},R(v)∩W≠∅R(v)=\{(0,0),(1,0)\},\quad W=\{(0,0)\},\quad R(v)\cap W\ne\varnothing
These sets are logical matrix locations for the declared shared-storage view and one write. An overlapping location establishes a potential dependency; proving the actual observable effect also uses the operation's values and order. The notation is not a complete byte-level alias analysis for arbitrary strided or overlapping tensors.↔ Scroll wide equations horizontally.

Functionalization changes representation, not required behavior

A functional representation can replace an update with a new tensor value and redirect later dependent uses. That can make dataflow easier to optimize, but externally observable mutations still need the required treatment at the function boundary. PyTorch 2.9 functionalization documents such fix-ups and limitations, including problematic global state and overlapping as_strided cases. Calling an intermediate graph functional does not authorize forgetting what callers see through an input alias after the original function returns.

MLIR bufferization approaches the opposite direction: it starts with tensor value semantics and chooses storage implementations. If an old tensor value remains needed after an operation that could overwrite its future buffer, the analysis must prevent that read-after-write conflict, potentially by copying. Safe in-place reuse depends on the old value's users and writability. It is not guaranteed merely because the new tensor has the same shape, or forbidden merely because it has a new SSA name.

sbefore=1+3=4,safter=(1+10)+3=14s_{\rm before}=1+3=4,\qquad s_{\rm after}=(1+10)+3=14
The sums use exact small integer values from the shared first-column view before and after the declared update. There is no concurrency in this example. Floating-point reordering, storage overlap beyond these locations, and external mutation would create additional obligations.↔ Scroll wide equations horizontally.

Build alias-positive and alias-negative fixtures

A useful test pair passes a view in one case and a real copy with the same initial contents in another. After mutating the base, the view must reflect the update while the copy need not. Also return or inspect the mutated input when that is part of the API contract; checking only a detached output can miss a dropped write-back. Add a nonoverlapping view case so a conservative optimization can be distinguished from an incorrect one, without requiring the compiler to optimize every provably safe opportunity.

The network lab illustrates only the distinction between evaluating an old value and a changed value. Reset, set both x=1,w=1, predict output about 0.6225, then Play or Step. Reset and set both x=2,w=1 for the changed-input output about 0.8176. Naming both results on paper prevents silently substituting one for the other. The lab has no shared tensor buffers, views, mutation, or functionalization pass, so the alias relationship itself must be established in a separate program fixture.

Detect an illegal hoist across a view mutation

X=[[1,2],[3,4]] owns mutable storage, v aliases its first column, and the program computes a=sum(v), then X[0,0]+=10, then b=sum(v), returning b−a. Use sequential exact arithmetic and no other writes.

  1. Before the update, v contains [1,3], so a=4. Retain that value as the result of the first read.

  2. The mutation changes X to [[11,2],[3,4]]. Because v is a shared view, its next read contains [11,3] and b=14.

  3. The specified return value is 14−4=10. Hoisting b's read before the update would instead produce 4−4=0, giving a concrete counterexample to that transformation.

The result

The update-to-read dependency is observable in the returned ten. Shape preservation and distinct SSA read results do not make the two reads interchangeable.

Explore the model

Interactive lab

Interactive lab · network

Keep old and updated evaluations distinct

Use fresh Reset → set both x=1,w=1 and Reset → set both x=2,w=1 comparisons. Predict sigmoid outputs ≈0.6225 and ≈0.8176 before Play or Step. This scalar instrument has no aliases or mutation and cannot validate a storage-reuse transformation.

Keep old and updated evaluations distinctOne inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Current readings: Product · wx: 1.2; Logit · wx − 0.5: 0.7; Sigmoid output: 0.6682. All parameters are adjustable with the labeled controls below.One input. One weight. One sigmoid.x1.20× w1.20+ b0.70σ0.668output0.668w = 1.00b = −0.50σ(z) = 1 / (1 + exp(−z))
Computation graphSignal trace
Product · wx
1.2
Logit · wx − 0.5
0.7
Sigmoid output
0.6682
t = 0 s

One inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA changed input can change a later read; real alias dependencies require explicit storage evidence.

Check your understanding

A functionalized graph has removed intermediate in-place updates, but the original function mutates an input observed by its caller. Which obligation remains?

Try it yourself

Repeat the worked example with v created as a genuine copy of the first column before either sum. The program still mutates X[0,0] by ten. What are a, b, and b−a, and what additional observation distinguishes this from dropping the mutation entirely?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

The copy does not share the updated location, but X is still externally mutable state.

Compare with the solution

The independent v remains [1,3], so a=4, b=4, and b−a=0. The base X must nevertheless become [[11,2],[3,4]]. Inspecting X after the call distinguishes correct copy behavior from an incorrect transformation that discarded the mutation. Equal return values alone are therefore not a sufficient oracle for this API: both the result and the promised state update belong to the test.

Take it with you

  • Aliasing depends on shared locations and access semantics, not just names or shapes.
  • A read cannot cross a relevant write without preserving its observed value.
  • Functionalization and bufferization must both preserve the original observable contract.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.