Keep integer timestamps attached to their rational timebases

A timebase is a rational number of seconds per tick. Multiply an integer timestamp by that rational value to obtain media time, keeping exact arithmetic as long as practical. PyAV v14.2.0 documents time units associated with streams, packets, frames, and containers; FFmpeg 7.1 headers define the relevant packet and frame timestamp fields. Do not assume a container-level microsecond offset and a stream tick count are interchangeable merely because both APIs accept an integer.

Choose a reference origin separately from the unit. Subtracting the first valid presentation timestamp can create a clip-relative timeline, but retain its mapping to the source timeline so annotations and other streams can be aligned. A nonzero or negative start is not automatically corrupt. Missing timestamps need a documented reject, repair, or inference policy with provenance; replacing them silently with frame index divided by nominal FPS converts uncertainty into fabricated timing. Also verify timestamp monotonicity in the presentation sequence rather than packet traversal order.

PTS describes presentation time, while DTS describes decoding time under the media API's semantics. A bidirectionally predicted picture can be presented between reference pictures even though the later-presented reference must be decoded first. Sorting encoded packets by PTS before feeding a decoder can therefore violate its input requirements. The decoder and demuxer contract governs packet consumption; the resulting frames and their timestamps govern temporal sampling for learning.

Variable frame intervals make a single reported average FPS insufficient for exact clip construction. Even a constant-rate source can have dropped, duplicated, edited, or discontinuous timestamps along a processing path. Measure differences between actual presentation times and preserve discontinuity markers. Presentation duration and elapsed decode time are also different: an offline decoder can process a minute of media quickly, while a live frame may arrive late. A causal system must track availability in addition to media position.

Store a manifest containing source identity, selected stream, timestamp/timebase pairs, chosen origin, decoding version, and any repair or resampling policy. Include a fixture with nonzero start time and unequal intervals so a frame-index shortcut cannot accidentally pass. Compare selected source timestamps, not only the number of output frames. When joining audio or labels, use their own units and origins explicitly rather than assuming every stream starts at the same tick value.

The modular lab illustrates a limited wrapped-counter ambiguity. Reset, set both modulus m=10 and additive step a=3, predict residues zero, three, six, nine, two, then take four Steps. Reset and set both m=10,a=1 for comparison. The last residue two in the first run corresponds to an unwrapped sum of twelve, showing that a displayed residue lacks an epoch count. This is not a video timestamp implementation and does not assert any particular container wrap rule; it isolates why units and reference information matter.