Event-time feature aggregation

Specify the entity key, event identifier, qualifying predicate, event timestamp, and precise interval. A ten-minute window ending at a prediction might include its lower boundary and exclude its upper boundary. An event exactly at prediction time then belongs to the next interval under that convention. Distinguish tumbling windows from overlapping sliding windows and continuously rolling features; a stored completed bucket is not automatically the exact rolling value at every request time. Document any approximation introduced by bucket resolution.

Out-of-order processing should not change membership in a deterministic event-time definition, although it can change which provisional result was available to a historic prediction. A late-known event can legitimately revise corrected history without having been visible at that original decision. Retain event identity for deduplication and capture revision or availability metadata for temporal retrieval. Replaying events in sorted event-time order is a useful oracle for final aggregate values, but does not reproduce the original publication timeline by itself.

Flink 2.1 uses watermarks to represent event-time progress and drive event-time windows. A multi-input operator ordinarily advances with its relevant input watermarks; an idle source can hold progress back unless handled under the documented idleness policy. Bounded-out-of-orderness generation encodes assumptions about observed timestamps, not a universal proof that an arbitrarily late producer can never emit older data. Watermark alignment and idleness have operational effects, so validate behavior with the actual source connectors and deployment.

Allowed lateness controls how long eligible window state remains available for late updates in the specified operator. A trigger determines when outputs appear, while accumulation mode determines whether repeated panes contain cumulative values or newly accumulated pieces. Beam exposes these distinctions explicitly. Flink’s late firings can update a previous window result. If the downstream online writer adds every cumulative firing as though it were a delta, it double-counts. Encode entity, window, definition version, and revision identity so repeated results can be replaced or otherwise combined correctly.

After a window’s retained state expires, the system needs a policy for later data: discard with metrics, route to a side output, rebuild affected buckets, or run a separate correction path. Silently allowing final offline recomputation to include all events while online serving permanently drops some creates a parity gap. Measure lateness distributions by source and entity class, because one global percentile can hide a producer with chronic delays. Keep enough source history to support whichever correction or audit policy you promise.

The queue instrument illustrates processing pressure rather than event-time semantics. Reset, set both λ=4 and μ=3, predict backlog six at t=2 from the initial four, then Step four times. Separately Reset, set both λ=2 and μ=3, predict drainage at t=4, then Step eight times. This deterministic fluid backlog has no event timestamps, watermark, window membership, or late firing. It demonstrates that a service-rate deficit can accumulate work; interpreting that count as a measured watermark delay would require information the model does not contain.