Record the dependencies a run actually creates

A job identifies a recurring computation or definition; a run identifies one execution of it. The same job can read different snapshots, execute different code versions, and produce different outputs on different days. Record those versions and the run's state instead of attaching one timeless edge to a mutable dataset name. OpenLineage models jobs, runs, and datasets with namespaced identities and extensible facets. A dataset reference alone does not necessarily pin its contents, so include the available snapshot, object version, or manifest identity required to reconstruct this particular execution.

Provenance also asks who or what was responsible for an artifact and activity. W3C PROV distinguishes entities, activities, and agents, supporting descriptions of derivation without implying that every input used by an activity contributed to every entity it generated. A computation may read a configuration for an unrelated diagnostic output while independently producing an embedding table. Turning every input and output into a Cartesian product of edges invents dependencies. Conversely, a dynamic lookup or external preprocessing service may be absent from static SQL analysis. Identify the capture mechanism and label declared plans, observed runs, and inferred relationships so users can assess which question each record answers.

Direct column lineage may say that a normalized feature is computed from a raw score using a specific transformation. Indirect influences include filters, joins, grouping, ordering, and conditional expressions that change which rows or values contribute. A permission flag used only in a WHERE condition can determine the training cohort even when it is not copied into the output. Omitting that edge makes it harder to audit the use boundary. Record the transformation category and scope that your lineage collector actually supports, and inspect how it handles aliases, user-defined functions, nested fields, and generated code.

Column lineage is not automatically a proof that a training example was eligible or a model was unaffected by a source defect. The captured run might not match the version used for training; the filter may have been coded incorrectly; the graph may omit side inputs. Keep code identity, policy version, dataset version, and run evidence connected. When a source field changes classification or is corrected, traverse verified dependencies to find candidate affected outputs, then validate actual membership and transformations. Over-approximated edges can support conservative investigation, but their false positives and missing coverage must remain visible.

Create small workloads whose dependencies are known: independent outputs, joins, filtered cohorts, renamed columns, and one dynamic side input. Check the collector's graph against these cases and retain its version and configuration. Capture loss, failed export, or unsupported operations should yield explicit uncertainty rather than an empty graph interpreted as no dependencies. A completed job event also does not prove that every output was atomically committed or that its consumer read that exact version. Link artifact publication and training execution records so the lineage can answer a concrete reconstruction question.

The probability lab provides a limited model of capture completeness. Reset, set both p=0.9 and n=10, predict the probability that every known edge is captured, then inspect the bar at ten successes. It is 0.9 raised to the tenth, about 34.9%, under independent equal capture probability. The lab neither extracts dependencies nor measures collector accuracy. Correlated unsupported operations can make the independence assumption especially poor. Use the calculation to challenge the idea that a high per-edge success rate automatically establishes a complete run graph, then return to deterministic known-case tests and actual export evidence.