advanced33 min studyLesson 699 in Engineering

Feedback needs stable identity, exposure attribution, and two separate notions of time

Prevent repeated delivery and late knowledge from changing recommendation labels silently while preserving distinct actions and the policy that exposed an item.

On this page

What you’ll understand

  • Distinguish delivery attempts, logical events, and impression-level labels.
  • Join feedback to the original exposure and policy rather than a current slate.
  • Apply both event-time and knowledge-time bounds in historical evaluation.

A click event can arrive twice, arrive late, or refer to a recommendation that was replaced before the collector received it. Counting deliveries as independent actions inflates feedback, while deduplicating by item can erase genuinely separate interactions. Attribution also needs the original exposure context: which request, item, position, policy, and propensity produced the opportunity. Finally, what happened by a particular time differs from what the system knew then. Reproducible labels and experiments need all three distinctions.

Choose an identity and an explicit label unit

Generate or preserve a stable logical event ID across retries and scope it so different producers cannot accidentally reuse the same key. A transport offset identifies a delivery location, which may differ from the user action's identity. Keep deduplication state for the promised horizon and apply its decision atomically with the relevant aggregate or state update when that guarantee is required. Duplicate suppression in one consumer does not imply that every downstream sink also applies an event only once, especially after replay or retention expiry.

Define whether the target is click count, unique clicked impressions, watch duration, or another outcome. Two valid clicks on the same impression can contribute two to an action count but only one to an at-least-one-click label. A repeated delivery should contribute neither an extra click nor a new impression. Similar timestamps or the same item ID are not sufficient proof of a duplicate. Schema and identity changes need versioned semantics so old and new producers do not silently apply different counting rules to the same evaluation window.

Nlogical=∣{key⁡(e):e∈D}∣,Yj=1{a qualifying unique click is attributed to impression j}N_{\rm logical}=|\{\operatorname{key}(e):e\in D\}|,\qquad Y_j=\mathbf1\{\text{a qualifying unique click is attributed to impression }j\}
D is a set or multiset of delivered records with valid stable scoped keys. N_logical counts logical events after key deduplication, while Y_j is one binary impression label under a stated qualification window. The formula does not solve identity errors or make every logical event a click.↔ Scroll wide equations horizontally.

Attribute to the exposure that actually occurred

Carry an impression or decision ID into feedback so it can join the original item, position, request context, eligible set, policy version, and recorded action probability. A join to the current recommendation list can attach an old click to the wrong slate. Define what qualifies as an impression: server delivery is not necessarily on-screen exposure or examination. Missing client acknowledgments and incomplete joins affect the denominator and can differ by device or network cohort. A label should preserve those limitations instead of converting every successful server response into certain user exposure.

Attribution rules are analytical choices. A delayed action can follow several exposures, and last-touch, first-touch, or a fixed-window rule can produce different credit assignments. They do not automatically identify which recommendation caused the action. For causal policy evaluation, preserve the action and outcome unit from the experiment or identification design. Record corrections and deletion effects in a way that can reproduce a historical dataset version. A convenient many-to-many join can otherwise multiply one outcome across several impressions and create artificial lift.

E[a,b),τ={e:a≤tevent(e)<b, tknown(e)≤τ}\mathcal E_{[a,b),\tau}=\{e:a\le t_{\rm event}(e)<b,\ t_{\rm known}(e)\le\tau\}
Event time assigns the real-world occurrence window [a,b), while known time is the defined first durable availability of that record for the historical query as of tau. All times share an interpreted clock convention. Corrections and versions need their own validity semantics; an event-time watermark alone is not a knowledge-time or causal guarantee.↔ Scroll wide equations horizontally.

Keep late knowledge from leaking into earlier decisions

A training snapshot claiming to represent what was known at an earlier cutoff must exclude events first available later, even if their action timestamps precede the cutoff. Otherwise features or labels can benefit from future knowledge. Conversely, a mature outcome dataset can intentionally wait for delayed feedback and use a later knowledge cutoff; document that horizon and the censored population. Watermarks express a stream processor's event-time progress policy, not proof that no late event will ever arrive. Choose allowed lateness, corrections, and reprocessing behavior explicitly.

The modular lab illustrates finite labels recurring, not an event-ID or deduplication service. Reset, set both modulus m=12 and additive step a=5, predict residues five after one Step and ten after two, then inspect. Reset and set both m=8 and a=2, predict orbit period four and return to residue zero after four Steps. Each Step advances one addition because it adds 0.5 seconds. The sequence is n×a mod m, starting at zero; it has no event records, time windows, joins, or collision-resistant identity. A recurring small numeric label cannot by itself distinguish arbitrarily many logical actions.

Three deliveries yield two logical clicks, only one known at the earlier cutoff

Click e1 belongs to impression i42, occurs at10:02, and is first durably known at10:10; it is delivered again at10:11 with the same ID. Click e2 belongs to i43, occurs at10:04, and is known at10:04. Both exposures and clicks satisfy the chosen attribution window.

  1. By10:12 there have been three deliveries but only two distinct logical click IDs, e1 and e2. Deduplicating by stable event identity prevents the replayed e1 from inflating the logical click count.

  2. Because the clicks attach to two different impressions, the mature at-least-one-click labels are Y_i42=1 and Y_i43=1. These are impression outcomes, not three independent clicks or evidence that either exposure caused its action.

  3. For event-time window[10:00,10:05) queried as known at10:05, only e2 is available. e1 occurred inside the window but arrived after the knowledge cutoff. Treating i42 as an established mature negative at10:05 would require an additional justified finalization rule, not merely absence of e1.

The result

Identity, attribution, event time, and knowledge time determine different parts of the dataset. Preserve them to reproduce both mature outcomes and the information available to earlier decisions.

Explore the model

Interactive lab

Interactive lab · modular

A recurring finite label is not a unique logical-event identity

Reset, set both m=12 and additive a=5, predict residues 5 and 10 after one and two Steps, then inspect. Reset and re-enter both m=8 and a=2 for period 4 and residue 0 after four Steps. The fixed orbit is n×a mod m from zero and advances one addition per 0.5-second Step. It has no event data, deduplication retention, attribution, timestamps, or identity guarantee.

A recurring finite label is not a unique logical-event identityAddition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Current readings: Residue r0: 0; Orbit period · m/gcd(m,a): 12; Distinct residues visited: 1 of 12. All parameters are adjustable with the labeled controls below.Repeated addition modulo 12: the current residue is 0, and the orbit period is 12.01234567891011+ 5mod 12An additive orbitn = 0rₙ = 0Period = 12Cycle starts at 0:0 → 5 → 10 → 3 →8 → 1 → 6 → 11 →4 → 9 → 2 → 7 → 0Exact residues · two additions per second
Additive orbitCurrent residue / next edge
Residue r0
0
Orbit period · m/gcd(m,a)
12
Distinct residues visited
1 of 12
t = 0 s

Addition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forChoose a stable scoped event identity and a retention protocol that matches the actual counting promise.

Check your understanding

A click occurred before a historical cutoff but was first available to the system afterward. Which snapshot rule prevents future-knowledge leakage?

Try it yourself

Four delivered click records have IDs x,x,y,z. Distinct events x and y both belong to impression p; event z belongs to impression q. Under retained stable-ID deduplication and an at-least-one-click-per-impression target, compute logical clicks and positive impression labels.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Deduplicate deliveries by event identity, then aggregate the distinct events at the label's unit.

Compare with the solution

There are three logical clicks, x,y,z. Impression p has a positive label despite two distinct click events, and q also has a positive label, giving two positive impression labels. The repeated x delivery adds nothing. This result assumes correct scoped identities, valid attribution, and events inside the chosen qualification and knowledge windows.

Take it with you

  • Delivery attempts, logical actions, and impression-level labels have different denominators.
  • Preserve the original exposure and policy when attributing feedback.
  • Event-time occurrence and historical knowledge cutoffs must both be explicit.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.