Engineering / advanced
Partition model computation across devices
Follow tensor dimensions, sequence activations, collectives, and placement through partitioned model layers while checking equivalent results.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Partition model computation across devices
Follow tensor dimensions, sequence activations, collectives, and placement through partitioned model layers while checking equivalent results.
- 01
Column parallelism: partition output features without summing them
Track matrix axes, local bias slices, and input-gradient reduction across a partitioned linear layer.
27 min - 02
Row parallelism: sum partial products before adding bias
Partition input features, reduce their contributions, and preserve nonlinear and bias semantics.
27 min - 03
Attention head partitions: keep each head’s dependencies intact
Track query and key/value head ownership, grouped-query constraints, and the output projection reduction.
27 min - 04
Sequence-parallel normalization: shard tokens, retain their hidden features
Explain why tokenwise LayerNorm stays local under sequence sharding and why its replicated parameter gradients still combine.
27 min - 05
Context parallel attention: exchange keys and merge softmax globally
Compute attention over remote token chunks using correct global normalization rather than averaging independently normalized outputs.
27 min - 06
Vocabulary shards: look up locally and normalize globally
Trace masked embedding ownership and a stable distributed softmax/cross-entropy without gathering every logit.
27 min - 07
Layout redistribution: a shard and a partial sum are different objects
Map DTensor placements to the required reconstruction and explain why identical local shapes do not determine semantics.
27 min - 08
Device meshes: multiply ranks without multiplying sharding claims
Compose tensor, data, context, and pipeline dimensions while accounting separately for parameter replicas and activation shards.
27 min - 09
Activation and traffic budgets: count lifetimes as well as shards
Estimate payload, temporary gather peaks, and declared collective traffic while distinguishing per-rank bytes from aggregate network work.
27 min - 10
Equivalence tests: reconstruct values, gradients, and the objective
Validate partitioned computation against a dense reference with deliberate shard boundaries, uneven tokens, masks, and numerical tolerances.
27 min