Engineering / advanced
Feed accelerators without changing the experiment
Design decoding, shuffling, sharding, caching, and restart state so that training input remains fast, representative, and reproducible.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Feed accelerators without changing the experiment
Design decoding, shuffling, sharding, caching, and restart state so that training input remains fast, representative, and reproducible.
- 01
Assign stable samples to explicit training owners
Distinguish dataset identity, traversal position, rank ownership, and tail handling so scaling workers does not silently change the training experiment.
24 min - 02
Randomize the stream you actually have
Analyze finite shuffle buffers, source ordering, seed scope, and sampling assumptions without mistaking reproducible order for representative training data.
24 min - 03
Scale the stage that limits batch production
Balance decoding, transformations, collation, and transfer while preserving sample and randomness semantics as input work becomes concurrent.
24 min - 04
Spend memory on enough ready work to hide bounded stalls
Budget buffered representations across ranks and workers while separating latency overlap from sustained production capacity.
24 min - 05
Pack tokens without changing which tokens can interact
Measure useful-token capacity and preserve attention, positions, targets, and loss weighting when replacing padded examples with packed sequences.
24 min - 06
Measure bytes fetched per useful training byte
Connect record layout, range requests, compression boundaries, and locality to remote-read cost while preserving dataset identity and sampling policy.
24 min - 07
Reuse transformations under a versioned semantic key
Place caches around expensive stable work while preserving random augmentation, artifact identity, and invalidation rules across training experiments.
24 min - 08
Make retries visible as training occurrences
Distinguish fetched, delivered, and optimizer-consumed work when workers fail, while preserving intended sampling multiplicities and explicit error policies.
24 min - 09
Restart from a consistent model and input boundary
Capture traversal, shuffle, random, and buffered state together with the training checkpoint so restart neither skips prefetched work nor invents an exact replay guarantee.
24 min - 10
Prove where a training step waits for its input
Use correlated host and device traces, controlled comparisons, and useful-work metrics to distinguish data starvation from compute, communication, and memory limits.
24 min