Budget the actual patch and tubelet sequence

For a clip with T frames and spatial size H×W, a nonoverlapping tubelet spanning τ frames and p×p pixels produces a product of temporal and spatial grid counts when all dimensions divide exactly. State whether the model pads, drops a remainder, overlaps patches, adds class tokens, or uses multiple pathways. A frame count also needs its temporal span from the sampling manifest. Doubling frames at fixed cadence lengthens the observed span, while doubling them in a fixed span increases sampling density; those are different information changes.

TimeSformer studies attention over frame-level patches, while VideoMAE uses a video masking and tubelet representation design. This lesson's generic tubelet formula should not be read as the exact input convention of every cited architecture. Record the actual implementation and any visible-token selection before estimating costs. An experiment that changes crop size, frame count, and masking simultaneously changes several resource and information axes, making a single memory or accuracy explanation difficult to isolate.

If full joint attention explicitly materializes one score tensor of shape [batch,heads,N,N], its storage is batch×heads×N²×bytes-per-score. Doubling N quadruples that tensor. This says nothing by itself about every activation, gradient, optimizer state, or fused kernel. An implementation that avoids materializing the complete score matrix can have a different memory footprint while still evaluating relevant attention interactions. Measure peak allocation under the actual training or inference mode.

Dividing temporal and spatial attention changes the interaction structure and its cost, rather than merely renaming the same full matrix. Masked pretraining can also run an encoder on a reduced visible-token set while a decoder and reconstruction path retain additional work. A high mask ratio therefore does not imply the whole training run becomes cheaper by the square of the visible fraction. Account for the components actually executed, including input decoding and data movement, before attributing end-to-end savings to a token count.

Use a small count oracle and an actual peak-memory measurement for representative maximum shapes. Test remainder dimensions and extra tokens so a formula does not silently omit a model's real sequence entries. Record whether precision casts create another copy and whether activation checkpointing changes retained state. A configuration that avoids memory exhaustion still needs a quality comparison: reducing frames or resolution can discard the evidence necessary for the task even when its allocation is perfectly understood.

The function lab isolates a quadratic scaling term in normalized units. Reset, set both coefficient a=1 and position x=1, predict value one, then Play or Step. Reset and set both a=1,x=2 for value four. Interpreting x as a token-count multiplier makes the squared growth visible, but the instrument contains no video tokens, attention heads, memory allocator, masking, or fused kernel. It validates only the stated quadratic arithmetic, leaving real-model peak memory and information sufficiency to separate checks.