Engineering / advanced
Collectives over accelerator fabrics
Compare collective algorithms and topology mappings using latency, bandwidth, ordering, congestion, and failure-aware communication models.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Collectives over accelerator fabrics
Compare collective algorithms and topology mappings using latency, bandwidth, ordering, congestion, and failure-aware communication models.
- 01
Collective semantics and participation
Specify the communicator, tensor contract, reduction meaning, and completion point before tuning distributed training communication.
25 min - 02
Ring AllReduce cost models
Derive ring phase counts and per-rank transfer volume while keeping latency, bandwidth, reduction work, and topology assumptions explicit.
25 min - 03
Tree and recursive-doubling collectives
Compare logarithmic step counts with transferred bytes and local reductions under explicit power-of-two and operator assumptions.
25 min - 04
ReduceScatter and AllGather decomposition
Track shard ownership, element counts, and layout when a full reduction is separated into distributed and replicated result phases.
25 min - 05
Hierarchical collectives and topology
Map local and cross-node phases onto real paths, shared links, and memory systems instead of treating accelerator connectivity as one uniform bandwidth.
25 min - 06
Communication-computation overlap
Construct the dependency and resource timeline before treating asynchronous submission as hidden communication time.
25 min - 07
Peer-to-peer transfer and ordering
Separate device addressability, matched communication, stream dependencies, and buffer ownership before treating accelerator-to-accelerator movement as complete.
25 min - 08
Collective deadlocks and mismatched calls
Diagnose rank-sequence, datatype, dependency, and failure mismatches without treating every timeout as insufficient network bandwidth.
25 min - 09
Fabric congestion and collective tail latency
Connect shared-link demand, rank arrival skew, and distributed maxima to training-step tails while keeping independence and traffic assumptions explicit.
25 min - 10
Collective microbenchmark interpretation
Read operation timing, algorithm bandwidth, normalized bus bandwidth, payload conventions, and correctness checks before attributing a result to hardware.
25 min