advanced27 min studyLesson 517 in Engineering

Fold what is known without inventing algebraic permission

Distinguish compile-time constants from dynamic inputs and preserve specified floating-point semantics when simplifying a partially evaluated tensor program.

On this page

What you’ll understand

  • Identify which input facts actually justify partial evaluation.
  • Construct a binary32 counterexample to unrestricted reassociation.
  • Separate constant evaluation from fast-math permissions and specialized-value guards.

Constant propagation computes facts from values the compiler genuinely knows. Partial evaluation uses those facts to perform part of a program early, leaving a residual computation for runtime inputs. Neither technique permits treating the first observed tensor contents as permanently fixed, nor replacing floating-point expressions with arbitrary real-number identities. A valid simplification depends on the declared input domain, operator semantics, and enabled numerical permissions. Make those premises visible so a smaller graph does not conceal a changed function or an unguarded parameter.

Known structure and known contents are different facts

A tensor's shape can be known while all its elements remain runtime values. Knowing that X has four columns allows shape arithmetic, but it does not justify precomputing XW when requests change X. A parameter may be intentionally frozen into a deployment artifact, supplied as a runtime input, or specialized with a checked value assumption. Each choice has a different update contract. If a model weight changes after capture, an artifact that embedded its old contents needs an appropriate invalidation or explicit immutable-artifact policy.

For a small real-valued expression f(x,c)=2(x+c), choosing c=3 leaves a residual expression depending on x. Evaluating the known c-dependent subcomputation is legitimate only under the chosen arithmetic semantics. A Python scalar captured as static can introduce a specialized-value condition, whereas a tensor scalar kept as an input remains dynamic. PyTorch 2.9 documents recompilations caused by changing scalar constants. The useful question is which values the artifact treats as fixed, not whether the source variable was given a suggestive name.

f(x,c)=2(x+c),f3(x)=f(x,3)f(x,c)=2(x+c),\qquad f_3(x)=f(x,3)
This is a real-valued mathematical specification with dimensionless scalars. It identifies specialization at c=3 without claiming that rewriting evaluation order to 2x+6 preserves every floating-point result. A runtime contract must additionally specify dtype and operation semantics.↔ Scroll wide equations horizontally.

Floating-point algebra carries counterexamples

Under binary32 round-to-nearest, ties-to-even, the integer 2²⁴ is exactly representable but the next integer is not. Adding one to it rounds back to 2²⁴. Subtracting the original value then gives zero. Regrouping the same written numbers can instead produce one, so associativity over real numbers is not an unconditional floating-point transformation rule. Constant evaluation should reproduce the semantics of the original operations unless the IR explicitly grants a broader numerical contract.

Other familiar identities fail on special values or signs. Replacing x−x by zero is wrong for an infinite x under ordinary IEEE behavior because the original produces NaN. Replacing zero times x by a positive zero also needs to consider NaNs, infinities, and signed zero. LLVM 20.1.0 describes fast-math flags that permit particular otherwise unsafe transformations; flags are contractual permissions and assumptions, not a proof that every result remains bit-identical. Preserve the actual flags rather than silently importing them from an optimization wish list.

fl⁡ ⁣(fl⁡(224+1)−224)=0,fl⁡ ⁣(224+fl⁡(1−224))=1\operatorname{fl}\!\left(\operatorname{fl}(2^{24}+1)-2^{24}\right)=0,\qquad \operatorname{fl}\!\left(2^{24}+\operatorname{fl}(1-2^{24})\right)=1
The example assumes binary32, rounding after each displayed arithmetic operation, round-to-nearest ties-to-even, and no extended-precision or reassociation substitution. fl denotes that rounding. The small values are dimensionless and show an exact counterexample under those premises.↔ Scroll wide equations horizontally.

Validate residual behavior across both value domains

Test a partially evaluated artifact against its reference using both the fixed specialization and deliberate changes that should invalidate it. For numerical rewrites, include cancellation, very small magnitudes, large finite values, and special values when the public domain permits them. Keep unsupported or explicitly excluded values separate from supported inputs. A tolerance-based deployment contract can be legitimate, but it must define what deviation is allowed and does not automatically excuse dropped branches, incorrect state, or stale embedded parameters.

The function lab illustrates a changed fixed coefficient in a real analytic expression, not floating-point compiler folding. Reset, set both coefficient a=0.5 and position x=2, predict value two and slope two, then Play or Step. Reset and set both a=1,x=2 for value four and slope four. These fresh evaluations show why a residual specialized at one coefficient cannot silently stand in for another. The instrument uses its own numerical implementation and is not a binary32 rounding emulator or a compiler pass.

Calculate a reassociation counterexample

Evaluate the two parenthesizations (A+1)−A and A+(1−A), where A=16,777,216. Use binary32 round-to-nearest ties-to-even after every operation, with no fused replacement or wider intermediate.

  1. A=2²⁴ is exactly represented. A+1 is halfway between adjacent binary32 values A and A+2, so ties-to-even rounds the first sum to A.

  2. The first expression therefore becomes A−A=0. In the second expression, 1−A=−16,777,215 is exactly representable in binary32.

  3. Adding A to that exact negative integer gives one, also exactly representable. The two operation orders therefore produce zero and one under the stated arithmetic contract.

The result

The expressions are algebraically equal over real numbers but not equivalent under this binary32 evaluation contract. A permitted relaxed contract must be stated separately.

Explore the model

Interactive lab

Interactive lab · function

Change a fixed coefficient and predict the residual

Compare fresh Reset → set both a=0.5,x=2 and Reset → set both a=1,x=2. Predict values 2 and 4 and slopes 2 and 4, then Play or Step. This analytic quadratic is not a binary32 emulator, constant-folding pass, or compiler execution.

Change a fixed coefficient and predict the residualThis lab uses only y = ax². Playback traces the curve within the fixed integration interval from 0 to x; it does not estimate numerical error or change the chosen point. Current readings: y at chosen x: 0.608; Tangent slope · 2ax: 1.35; Signed area · ax³/3: 0.182. All parameters are adjustable with the labeled controls below.−2.4−1.201.22.4−9−4.504.59xy(x, ax²)
y = ax²TangentSigned integral
y at chosen x
0.608
Tangent slope · 2ax
1.35
Signed area · ax³/3
0.182
t = 0 s

This lab uses only y = ax². Playback traces the curve within the fixed integration interval from 0 to x; it does not estimate numerical error or change the chosen point. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA specialization depends on the actual fixed value and the arithmetic contract.

Check your understanding

A compiler knows an input tensor has shape [8,4] but its elements vary across requests. Which simplification follows from that fact alone?

Try it yourself

An artifact has embedded c=3 for the real-valued function f(x,c)=2(x+c). At x=4 it returns fourteen. A later request supplies c=5 with the same tensor shape. What should the reference return, and what must protect reuse of the c=3 artifact?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Evaluate the declared function using the new value, then separate a legitimate specialization from a dynamic-argument implementation.

Compare with the solution

The reference value is 2(4+5)=18, while the old specialization returns 2(4+3)=14. If c is specialized, dispatch must establish c=3 or select a correct alternate route; if c is dynamic, the generated program must read the new input. Matching shapes do not justify reuse of embedded values. This exact small-number example isolates stale specialization without relying on floating-point tolerance arguments.

Take it with you

  • Known dimensions do not make tensor contents constant.
  • Partial evaluation must retain the fixed-value and update contract.
  • Floating-point folding and reassociation must respect the IR's actual numerical permissions.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.