Engineering / advanced
Engineer a controlled pretraining run
Connect objective definition, token accounting, initialization, schedules, validation, and recovery to a reproducible autoregressive training experiment.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Engineer a controlled pretraining run
Connect objective definition, token accounting, initialization, schedules, validation, and recovery to a reproducible autoregressive training experiment.
- 01
Define the next-token task before counting a training example
Specify shifted targets, valid-loss positions, document boundaries, and the conditioning context of a reproducible autoregressive objective.
28 min - 02
Freeze the tokenizer as part of the experiment
Balance vocabulary parameters, sequence lengths, normalization, corpus sampling, and held-out separation without mistaking token-level loss for a tokenizer-independent metric.
28 min - 03
Pack documents without changing who can attend to whom
Construct independent-document causal blocks, shifted loss masks, position policies, and isolation fixtures for efficient sequence packing.
28 min - 04
Count the tokens that actually contribute to each update
Derive global valid-token batches and loss normalization across ranks and accumulation microbatches without silently weighting short batches more heavily.
28 min - 05
Use variance arithmetic as a scoped initialization model
Analyze linear fan-in and residual covariance before turning an initialization rule into a claim about deep Transformer stability.
28 min - 06
Define the schedule clock and every endpoint
Specify update indexing, warmup, cosine decay, skipped steps, and restored scheduler state so the learning rate applied to an update is reproducible.
28 min - 07
Treat a loss spike as evidence to investigate
Preserve batch, numerical, optimizer, and infrastructure evidence before attributing a spike, and distinguish gradient clipping from a stability guarantee.
28 min - 08
Compare validation loss on a fixed, counted evaluation contract
Decompose token-weighted domain loss, derive perplexity correctly, and keep tokenizer, context, masks, and evaluation mixture stable.
28 min - 09
Track cumulative exposure when the domain mixture changes
Define token-based mixture schedules, distinguish sampling probabilities from realized exposure, and audit repetition against available unique data.
28 min - 10
Accept a pretraining artifact with reproducible evidence
Connect configuration and data manifests to objective fixtures, fixed evaluation, complete checkpoint state, and a demonstrated recovery drill.
28 min