Data pipelines and lineage
Build reproducible training datasets by tracking identities, event time, transformations, and publication boundaries.
On this page
The question for today
What you’ll understand
- Distinguish event time, processing time, and dataset publication time.
- Design a replayable pipeline with explicit lineage and quality gates.
- Calculate backlog recovery without confusing throughput with freshness.
A training job can finish successfully while learning from yesterday's wrong join. Data infrastructure must explain which records existed, which transformations ran, and which version a model actually consumed. Imagine a recommendation model trained on purchases and product descriptions: a late purchase, a corrected description, and a duplicate event are different changes. A useful pipeline gives each a deliberate meaning rather than silently overwriting a directory called latest.
Turn a collection of files into a reproducible artifact
Start with an immutable input identity: source, partition or object version, schema version, and a stable event identifier. A transformation identity includes code revision, configuration, dependency versions, and any random seed. A run is one execution of that transformation. These distinctions let an engineer separate a changed algorithm from a retry of the same algorithm. OpenLineage's dataset, job, and run entities provide one standard vocabulary for recording those relationships.
A dataset release should name a manifest containing exact output objects and validation results. Consumers read the release identity, rather than discovering files while producers are still writing them. Retain the mapping from training run to release and from release to inputs. A content hash detects byte changes, but it cannot establish licensing, consent, label correctness, or freedom from training–test leakage. Record those checks as explicit evidence with owners and expiry policies.
Decide what late data means
Event time describes when a business event occurred; processing time describes when this system handled it. A purchase made at 23:58 may arrive after midnight. Grouping solely by arrival date moves it into the wrong business window. A watermark is an estimate of progress in event time, not proof that earlier events can never arrive. Define a lateness policy: revise an existing window, produce a correction, or quarantine records beyond an agreed horizon.
For point-in-time training features, join each example to information available at its prediction time. Joining to the newest customer profile can leak future behavior even when every timestamp parses correctly. Backfills must follow the same temporal rule as live production. Keep raw events where policy permits, deduplicate by a stable business key, and make repeated publication of the same logical partition harmless. Exactly reproducing corrupted inputs is reproducibility, but it is not correctness.
Measure freshness and recover safely
Throughput measures work per second; freshness measures how old the usable information is. Monitor input age, oldest unprocessed event, invalid-record share, and time since the last valid publication. A pipeline producing many rows can still be stale if it replays the same partition. Conversely, an idle source is not necessarily broken. Compare freshness to source behavior and a declared service objective before paging someone for an empty queue.
For a fluid backlog model, arrivals continue while workers drain the queue. If sustainable processing rate exceeds arrival rate , recovery time is backlog divided by spare throughput. Rates must use the same record definition, since filtering and fan-out change counts between stages. Real recovery also includes partition skew, retries, and storage limits. Test replay into a separate version, compare counts and feature distributions, then publish only after validation; do not let a backfill compete unboundedly with current data.
Work it through
Recover a feature pipeline after an outage
An ingestion stage has 18 million queued events. Live traffic is 20,000 events/s and validated processing capacity is 50,000 events/s. Can it recover within 12 minutes?
Use spare throughput, not total throughput: 50,000 − 20,000 = 30,000 events/s are available to reduce the backlog.
Divide the 18,000,000-event backlog by 30,000 events/s. The ideal drain time is 600 seconds, or 10 minutes.
Allow 90 seconds for the final join, validation, and manifest publication. The estimated usable-data recovery becomes 690 seconds, or 11.5 minutes.
The 30-second margin is small. Confirm that hot partitions and the database can sustain the measured rate, and throttle backfill if current-data freshness worsens.
The constant-rate model meets 12 minutes, narrowly. Publication and quality checks belong in the recovery estimate, not just queue draining.
The living notebook
Explore the model
Interactive lab · queue
Watch a backlog form and drain
Vary arrival λ and service μ in a fluid backlog max(0, 4 + (λ − μ)t). Interpret items as events at one stage. The separate M/M/1 mean assumes Poisson arrivals and exponential single-server service; neither model simulates event-time watermarks or lineage.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forMore processing than incoming work is necessary to catch up; zero backlog alone does not prove a correct dataset.
Pause & reason
Check your understanding
Your turn
Try it yourself
A stage has 9 million queued records, arrivals of 15,000 records/s, and capacity of 25,000 records/s. Publication adds 60 seconds. Estimate usable recovery time and explain what happens if arrivals rise to 26,000 records/s.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Only capacity left after live arrivals reduces the backlog. Check the sign of that spare capacity before dividing.
Compare with the solution
Spare capacity is 10,000 records/s, so draining takes 900 seconds. Publication adds 60 seconds, giving 960 seconds, or 16 minutes. At 26,000 arrivals/s the backlog grows by 1,000 records/s; the constant-capacity model has no finite drain time. Additional capacity or controlled admission is required.
Keep the idea
Take it with you
- Version inputs, transformations, and published outputs separately.
- Late-arrival policy and point-in-time features are part of data correctness.
- Recovery depends on spare capacity and time to validated publication.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.