Representation learning and contrastive losses

Two augmented views of one image are often treated as a positive pair. Cropping, color changes, and blur then declare which transformations should leave the representation similar. This is appropriate only when those transformations preserve the relevant semantics. A crop can remove the only diseased region in a medical image; changing color can erase a color-based label. The augmentation policy is therefore part of the task specification, not merely a way to generate more training rows.

An encoder maps an input to a representation, and a projection head maps that representation into the space used by the contrastive loss. Unit normalization makes dot products into cosine similarities. A projection head can absorb loss-specific constraints while leaving an earlier representation useful for transfer. Decide which layer will be evaluated downstream. Measuring only the projection output can answer a different question from measuring the encoder representation used by a deployed classifier.

A positive score competes with all included candidates through the softmax denominator. Lower temperature sharpens this competition and increases sensitivity to score differences, but it also changes gradient scale and can amplify difficult or mislabeled comparisons. Larger batches provide more in-batch candidates, with greater memory and communication costs. Stored representations from a memory bank can add candidates but may have been encoded by older parameters.

A negative example is not guaranteed to be semantically unrelated. Two different images of the same category can become false negatives if the objective pushes them apart. Duplicate records and correlated observations can also make candidate counts overstate diversity. Distributed training must define whether embeddings are gathered across devices and how gradients through that gather behave. These details change the effective loss; matching only local batch size does not reproduce a global candidate set.

A frozen linear probe measures how accessible labels are to a linear classifier with an unchanged encoder. Fine-tuning measures a different capability: how well the representation adapts when all or some parameters may move. Retrieval evaluation depends on the candidate collection, duplicate handling, and relevant-item definition. Report these protocols separately. A lower pretraining loss need not imply better results under every probe or domain shift.

Keep validation and test identities separated when records contain repeated people, products, or documents. Inspect nearest neighbors for shortcuts such as watermarks or acquisition devices. Pairwise similarity matrices require quadratic storage in the number of candidate embeddings if materialized naively; chunking can reduce peak memory while retaining the stated objective. Any approximation, including sampled negatives, should be reported with its sampling distribution. The representation's geometry is a product of both data construction and computation.