Transform a clip and its labels under one explicit rule
Preserve coherent spatial transforms across a clip while checking which label meanings survive cropping, reflection, temporal reversal, and speed changes.
On this page
The question for today
What you’ll understand
- Distinguish label invariance from a required annotation transformation.
- Apply one declared geometric convention across a clip and its boxes.
- Reject augmentation policies that erase or reverse the labeled evidence.
Augmentation creates another training view, but it also changes the evidence and sometimes the answer. A crop applied independently to every frame can invent apparent camera motion. A horizontal reflection can reverse a direction label, and temporal reversal can change opening into closing. Consistency therefore has two parts: the geometric or temporal transformation must be coherent under its declared policy, and the label must remain valid or be transformed accordingly. Neither part follows automatically from an API accepting a video tensor.
State which transformation is shared across time
For a spatially consistent clip crop, sample its parameters once and apply the same coordinate mapping to every selected frame. Specify whether resizing, reflection, and color changes share parameters or intentionally vary over time. TorchVision 0.24 documents video-capable transforms and explicit functional operations; inspect the actual call structure and data types rather than assuming separate per-frame calls share a random draw. A reproducible manifest records the sampled parameters and seed policy, not only the transform's class name.
A fixed crop removes inconsistent artificial shifts from that crop operation, but it does not guarantee preservation of all physical motion or task evidence. The object can leave the field of view, resizing can erase small details, and a temporal subsample can already have aliased the action. Evaluate whether the transformed clip still contains sufficient evidence for the intended label. Consistency is a property of the transformation rule, whereas validity depends on the task and the particular sampled content.
Carry coordinates and semantics through the same mapping
Declare a box convention before applying arithmetic. Continuous edge coordinates with half-open intervals differ from integer pixel-center formulas, especially for reflection and endpoints. Crop by subtracting the crop origin, intersect with the valid crop extent, then scale and reflect under the chosen order. Clip or discard empty boxes according to a documented policy, and transform masks, keypoints, or tracks using their compatible coordinate rules. Merely keeping the original annotation beside a transformed tensor is not label preservation.
Semantic labels can require changes even when coordinates remain well defined. A leftward-motion class can become rightward after horizontal reflection, while a nondirectional object class might remain unchanged if its evidence survives. Temporal reversal can invalidate causal event ordering and speed augmentation can invalidate a rate label. Do not infer class invariance from common benchmark preprocessing. Keep a task-specific allowlist and negative fixtures, such as an opening/closing pair, so unsupported transformations cannot silently enter training.
Test coherence and label validity with separate oracles
Build a synthetic moving box with known source coordinates and timestamps. Verify each transformed frame and annotation against the same mapping, including border intersections and empty targets. Then check a semantic fixture where a direction or event-order label must change. A seeded rerun should reproduce the selected view without making distinct clips receive accidental identical randomness. Record library versions and the sampled operations so later changes to defaults do not masquerade as model improvements.
The matrix lab offers a limited coherent spatial mapping. Reset and set both a=1,b=1, predict determinant one, then Play or Step; Reset and set both a=2,b=1 predicts determinant two. Its selected matrix is [[a,b],[0,1]], and readouts describe that final matrix while geometry interpolates. Every supported a is positive, so this family cannot perform horizontal reflection, cropping, or label transformation. It illustrates applying one stated linear map rather than proving that an augmentation preserves video semantics.
Work it through
Transform a box through crop, reflection, and resize
A fictional 100×80 image uses continuous edge boxes [x₁,y₁,x₂,y₂]. A box is [20,10,60,50]. Apply crop [10,0,90,80], then horizontal reflection inside that 80×80 crop, then resize to 160×160. The box remains fully visible.
Subtract the crop origin (10,0): the box becomes [10,10,50,50] in the 80×80 cropped coordinate system. No clipping is needed for this particular box.
Reflect the horizontal edges with W=80: [x₁,x₂]=[10,50] becomes [80−50,80−10]=[30,70]. The full box is [30,10,70,50].
Resize by a factor of two on both axes to obtain [60,20,140,100]. Use that transformed box with the transformed images, then separately apply any required semantic label change such as reversing a directional class.
The resulting continuous-edge box is [60,20,140,100]. This geometric result does not establish that every class label is invariant to the reflection.
The living notebook
Explore the model
Interactive lab · matrix
Apply a declared map without claiming a full augmentation engine
Fresh Reset → set both a=1,b=1 predicts det(A)=1; Reset → set both a=2,b=1 predicts det(A)=2. Play interpolates geometry, but readouts describe selected A. The supported positive-scale shear family cannot reflect, crop, transform tracks, or validate semantic labels.
- Area scale · det A
- 1.4
- Singular values · σmax, σmin
- 1.597, 0.876
- Condition number · κ₂(A)
- 1.823
The selected matrix is A = [[a,b],[0,1]]. Playback interpolates geometry between the identity and A; readouts always describe A. Its determinant is a and its singular values come from AᵀA. Positive a makes every supported matrix invertible. This two-dimensional family is not a general eigensolver or a learned network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forCoherent geometry and valid labels require explicit, separate checks.
Pause & reason
Check your understanding
Your turn
Try it yourself
An original box occupies horizontal interval [90,100), and the crop retains original horizontal interval [10,80). What is the intersection width? Can a positive 'object visible in clip' target be retained solely because the original annotation was positive?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Intersect the intervals before converting coordinates. An empty geometric target cannot by itself support a visibility label.
Compare with the solution
The intersection width is max(0,min(100,80)−max(90,10))=0. The object box is outside this crop. The pipeline should follow its declared discard, relabel, or resample policy; it cannot claim visibility simply by copying the source label. If other evidence legitimately supports a different clip-level task, that needs a separate semantic rule rather than an unchanged box annotation.
Keep the idea
Take it with you
- Sample shared geometric parameters deliberately and record the resulting view.
- Coordinate conventions and transform order determine the correct annotation arithmetic.
- A coherent augmentation can still remove evidence or change a label's meaning.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.