Vision transformers and multimodal interfaces

For an H by W image with C channels and nonoverlapping square patches of side P, assume both spatial dimensions are divisible by P. Each patch flattens to P²C numbers and is projected to model width d. The sequence contains HW/P² patch tokens, possibly plus a class token. Position representations distinguish patch locations because the flattened patch collection alone does not encode the original grid order. Padding or cropping policies must be explicit when images do not divide evenly.

The patch projection is a learned linear map, often implemented as a convolution with kernel and stride both equal to the patch size. This implementation equivalence does not make the full architecture a conventional convolutional network: later token mixing and parameter sharing differ. Large patches reduce sequence length but compress larger regions at the input interface. Fine spatial details may be harder to recover after that compression, particularly for localization tasks requiring more than a global class label.

A dual encoder maps an image and a text independently into a shared embedding space. A similarity score then supports retrieval or classification against candidate text descriptions. Image embeddings can be precomputed because they do not depend on the current text query. Contrastive training aligns matched pairs relative to negatives, but it does not by itself provide a token-level explanation of which word corresponds to which region or a mechanism for generating an answer.

A fusion model instead permits tokens from one modality to influence tokens from another, through joint self-attention or cross-attention. An image representation can then depend on the question, enabling more detailed interactions at additional computation cost. In cross-attention, query length and key-value length may differ. A system with an image encoder, a projector, and a language decoder must train or otherwise specify the interface between feature spaces; matching vector dimensions alone does not align their meaning.

Doubling both image dimensions at fixed patch size quadruples the token count. A dense attention score matrix then has sixteen times as many entries per head, though matrix projections and other operations scale differently. Fused attention implementations can avoid storing the full score matrix while retaining dense arithmetic. Resolution policy therefore affects memory, latency, and accessible detail. Report actual resized dimensions, aspect-ratio handling, and any token selection instead of just naming the backbone.

Multimodal evaluations should test whether the model uses the intended modality. Compare against text-only and image-only baselines, perturb irrelevant content, and test counting or spatial relations when those capabilities matter. Training image-text pairs may contain noisy captions, duplicates, or sensitive correlations. A strong retrieval average does not prove compositional reasoning, and an answer that sounds fluent does not prove visual grounding. Evaluate the complete encoder-interface-decoder path on deployment-style inputs.