Engineering / advanced
Automatic differentiation as a runtime system
Build derivative transformations from vector products, saved state, control flow, mutation rules, and recomputation rather than treating gradients as an oracle.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Automatic differentiation as a runtime system
Build derivative transformations from vector products, saved state, control flow, mutation rules, and recomputation rather than treating gradients as an oracle.
- 01
A JVP propagates one input direction through the program
Track primal values and tangents, calculate a shaped Jacobian-vector product, and distinguish directional differentiation from finite differences and a full Jacobian.
25 min - 02
A VJP pulls an output sensitivity back to the inputs
Calculate cotangent shapes, seed a non-scalar output, accumulate fan-out contributions, and distinguish a reverse product from a full Jacobian.
25 min - 03
Reverse execution needs the forward values its rules depend on
Trace a small reverse tape, distinguish graph structure from saved storage, and reason about activation lifetimes, aliases, and memory accounting.
25 min - 04
A custom rule must be a compatible linear derivative map
Specify primal domains, tangent and cotangent shapes, seed linearity, higher-order support, and explicit surrogate conventions before overriding differentiation.
26 min - 05
A saved tensor needs its historical value, not merely its storage address
Trace mutation hazards through aliases, distinguish detach from copying, and explain version checks without assuming every in-place operation is forbidden or safely detected.
25 min - 06
The executed branch has a derivative only under local path assumptions
Trace conditional and loop differentiation, distinguish piecewise derivatives from boundary conventions, and avoid undefined operations hidden behind a later mask.
26 min - 07
Differentiating a derivative needs its own graph
Calculate a Hessian-vector product, distinguish derivative-graph construction from retaining saved context, and state smoothness and custom-rule limitations.
26 min - 08
Checkpointing trades saved activations for compatible recomputation
Budget a simple segmented reverse schedule, preserve forward semantics across RNG and device behavior, and distinguish ideal memory counts from measured peak cost.
28 min - 09
Repeated sparse indices accumulate before nonlinear updates
Derive a gather's scatter-add adjoint, distinguish duplicate storage entries from gradient identity, and budget sparse values and indices under supported runtime layouts.
26 min - 10
Derivative checks provide targeted evidence, not an oracle
Choose directional finite-difference scales, test adjoint consistency with shaped seeds, and separate smoothness, rounding, stochastic replay, and mutually wrong derivative rules.
28 min