Engineering / research
Attention algorithms and data movement
Derive exact tiled attention and distinguish its I/O savings from arithmetic complexity before evaluating sparse or approximate alternatives.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Attention algorithms and data movement
Derive exact tiled attention and distinguish its I/O savings from arithmetic complexity before evaluating sparse or approximate alternatives.
- 01
Merge normalized attention without averaging tile outputs
Derive stable running summaries, their rescaling law, and an explicit empty-row contract.
26 min - 02
Save memory traffic while retaining dense arithmetic
Separate dense matrix work, materialized intermediates, on-chip tiling, and the scope of I/O asymptotics.
26 min - 03
Recompute probabilities and differentiate the normalized row
Derive softmax row gradients, saved normalization state, and the conditions for consistent backward recomputation.
26 min - 04
Preserve sequence identity and global positions across tiles
Derive packed causal eligibility, unequal-length alignment, and empty-row behavior from logical positions.
26 min - 05
Share keys and values without confusing query-head identity
Derive GQA head mapping, cache payload, shared gradients, and its distinction from changing a kernel schedule.
26 min - 06
Count active blocks and acknowledge the changed operator
Derive block-pair work, distinguish block sparsity from element masks, and test against the matching sparse reference.
26 min - 07
Count local and global connections without double-counting
Derive window edge counts, global-token unions, and multi-layer routes under explicit boundary assumptions.
26 min - 08
Reassociate a feature-map kernel without claiming exact softmax
Derive feature aggregation, positive normalization, causal prefixes, and the distinction between a new kernel and softmax approximation.
26 min - 09
Test the operator, its derivatives, and its scaling separately
Build deterministic reference cases, meaningful error metrics, metamorphic checks, and mechanism-aware complexity experiments.
26 min - 10
Benchmark an architecture-specific schedule within the whole workload
Separate mathematical portability from kernel support, hardware-specific scheduling, and end-to-end performance.
26 min