Attention and the Transformer computation

Each input representation is projected into a query, a key, and a value. A query compares with keys to produce scores; softmax turns an allowed row of scores into nonnegative weights summing to one; those weights mix the values. A key is therefore used for matching, while a value carries the content being combined. They can be learned from the same input without being the same vector. Multiple heads perform different learned projections and their outputs are combined.

Dividing dot-product scores by the square root of key dimension controls their scale under a rough independent-component variance argument. It is not a probability-calibration theorem. Position information is needed because content-only attention does not by itself encode a sequence's order. Different architectures represent positions differently; the original Transformer used sinusoidal encodings. Attention weights describe one internal mixing operation, but should not automatically be interpreted as a faithful explanation of the model's final decision.

An autoregressive model predicting the next token must not read future target tokens. A causal mask allows a position to read its own representation and earlier positions but excludes later ones. During teacher-forced training, the real prefix is available and the target is shifted to the following token. A wrongly shifted target or incorrectly oriented triangular mask can make loss look impressive by allowing the answer to enter its own prediction. Verify a tiny sequence where every allowed dependency is visible.

Padding masks serve a separate purpose: padding is not real context and should not contribute as a key. Ensure every computed attention row has an intended valid source; a softmax over only negative infinities is undefined in ordinary arithmetic. Library mask conventions are not universal. For example, a boolean mask's meaning can differ between attention interfaces, so check the exact API rather than copying a tensor from another function. Test mask behavior through changed inputs, not shape checks alone.

Dense attention computes a score for every allowed query-key pair, giving quadratic pairwise work in sequence length for the standard dense formulation. Doubling context can therefore be much more expensive than doubling a batch dimension. Memory-efficient implementations can avoid storing the entire score matrix without changing the intended exact attention result up to floating-point differences; that does not erase all pairwise arithmetic. Other attention patterns trade different communication structures and approximation assumptions against cost.

An attention layer alone is not a language model. Tokenization, embeddings, normalization placement, residual connections, feed-forward layers, the training objective, and data all influence behavior. During generation, cached keys and values avoid recomputing some prefix work but consume memory that grows with context. Investigate a model with small controlled examples: changing a masked future token should not alter an earlier position's causal prediction. An architectural diagram becomes useful when it suggests a falsifiable behavior check.