Engineering / advanced
Design and measure tensor kernels
Derive kernels from tensor layouts and reuse, then validate reductions, matrix multiplication, attention, and fusion against numerical references.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Design and measure tensor kernels
Derive kernels from tensor layouts and reuse, then validate reductions, matrix multiplication, attention, and fusion against numerical references.
- 01
Tensor strides: prove which element an address represents
Derive storage addresses for views, transposes, padding, and broadcasts before choosing a contiguous or strided kernel path.
24 min - 02
Tiled matrix multiplication: assign outputs and count useful work
Derive matrix tiles, accumulation, edge masks, and conventional FLOP counts before comparing a kernel with a transfer-bound performance ceiling.
24 min - 03
Shared-memory reuse: stage a tile and prove who may overwrite it
Calculate block-level operand reuse while respecting synchronization, staging capacity, and the difference between logical loads and device-memory traffic.
24 min - 04
Tensor Core operands: a matrix instruction still needs a layout contract
Separate matrix shape, memory layout, fragment mapping, operand precision, and accumulation precision in an architecture-scoped Tensor Core path.
24 min - 05
Reduction trees: coordinate the work and specify the numerical answer
Design a parallel sum with correct participation, edge identities, cross-block combination, and an explicit floating-point accuracy contract.
24 min - 06
Stable softmax: keep the normalizer consistent as tiles arrive
Derive max-shifted softmax, combine online block summaries, and connect numerical state reuse to attention without materializing every score.
25 min - 07
Normalization kernels: compute the right statistics and reuse the row
Specify LayerNorm axes, population variance, epsilon placement, parallel moment combination, and resource-limited reuse.
24 min - 08
Transpose kernels: coalesce the boundary and inspect bank mapping
Separate a transpose view from a physical layout conversion, then derive a scoped shared-memory padding example and its costs.
24 min - 09
Kernel fusion: remove traffic without hiding resource costs
Compare materialized and fused pointwise stages while accounting for intermediate rounding, registers, spills, occupancy, and other consumers.
24 min - 10
Kernel harnesses: verify the operation before timing its boundary
Build reference cases, numerical tolerances, memory-safety checks, warmup, synchronized timing, and unit-consistent reports for tensor kernels.
25 min