Engineering / advanced
Experiment with models in live systems
Analyze assignment, logging, feedback delay, interference, and rollout decisions while distinguishing causal online evidence from offline score differences.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Experiment with models in live systems
Analyze assignment, logging, feedback delay, interference, and rollout decisions while distinguishing causal online evidence from offline score differences.
- 01
Randomize the entity whose outcome you intend to compare
Connect stable assignment, repeated requests, analysis units, and the causal question.
25 min - 02
Keep eligibility, assignment, and actual intervention separate
Design a joinable exposure ledger without conditioning causal estimates on treatment-created activity.
25 min - 03
Randomize groups when one treatment changes another unit’s outcome
Distinguish direct assignment effects, global rollout effects, cluster designs, and dependence-aware uncertainty.
25 min - 04
Wait for comparable outcome windows or model censoring explicitly
Separate event time, follow-up maturity, attribution, and logging lag in model experiments.
25 min - 05
Use allocation imbalance to investigate the observation process
Calculate an SRM diagnostic and distinguish detection, root cause, and causal validity.
25 min - 06
Monitor with a rule whose error guarantee survives stopping
Contrast naïve repeated fixed-horizon tests with predeclared multiple looks and valid confidence sequences.
25 min - 07
Remove predictable variation without adjusting away treatment
Derive CUPED control variates, their variance reduction, and the assumptions that keep the treatment contrast intact.
25 min - 08
Estimate a policy from logged actions only where identification permits
Derive inverse propensity and doubly robust estimators with explicit support, observation, and nuisance-model assumptions.
25 min - 09
Choose between learning an effect and allocating reward
Separate cumulative-reward objectives, changing propensities, nonstationary mixtures, and valid adaptive inference.
25 min - 10
Decide with uncertainty and retain a compatible rollback state
Combine effect bounds, guardrails, data-quality checks, and immutable serving manifests without overstating evidence.
25 min