Attention is a way of selecting information in deep learning: when a model processes one position in a sequence, it does not treat all other positions equally. It computes a weight for each and pulls information in according to those weights. High-weight positions strongly shape the result; low-weight ones barely register.
The idea first appeared in machine translation in 2014, to fix the problem of long sentences losing their beginnings. In 2017, "Attention Is All You Need" pushed it to its conclusion: drop the recurrent structure entirely and keep only attention. Every large model's compute cost, context length limit, and caching mechanism traces back to this one component.
Grab It in One Sentence First
Attention is the model deciding for itself which parts of the input matter most at this particular step.
Picture reading a sentence: "Zhang handed the file to Li, because he was in a hurry." To work out who he is, your eyes flick between "Zhang" and "Li" rather than re-reading every character. Attention does exactly that, except "who to look back at, and how hard" becomes a set of numbers that can be learned.
How It's Computed
The standard formulation has three roles: Query, Key, and Value. Every position projects all three out of its own representation.
A library analogy: the Query is the question you came in with, the Key is the subject label on each book's spine, and the Value is what is actually inside the book. You compare your question against every label, then blend the books' contents in proportion to how well they matched — relevant books contribute a lot, irrelevant ones almost nothing.
Self-attention means Query, Key, and Value all come from the same sequence, so words in a sentence look at each other. Cross-attention lets one sequence look at another — for instance, letting a generated image attend to a text prompt.
What "Multi-Head" Means
Real models don't compute one attention pattern but several in parallel — multi-head attention. Each head uses different projection matrices and learns a different pattern. Interpretability work has observed heads that consistently attend to the previous token, heads that track matching punctuation or brackets, and heads doing coreference resolution. The heads' outputs are concatenated and passed through a linear layer.
The intuition is that one position relates to others in more than one way. With a single set of weights, syntactic and semantic relationships would be crammed into the same numbers and interfere with each other.
Why It Sets the Cost of the Context Window
For a sequence of length n, standard self-attention computes n×n relevance scores. Double the input and that computation roughly quadruples, while the intermediate state that must be cached grows linearly. That is the fundamental reason context windows can't simply be made unlimited, and why long documents are noticeably slower and pricier.
An entire engineering field grew around this bottleneck:
- KV caching. During generation, Keys and Values already computed for earlier tokens are reused rather than recomputed. This is the mechanism behind prompt caching savings, and why putting a stable system prompt first pays off.
- Grouped-query attention (GQA/MQA). Multiple Query heads share one set of Keys and Values, sharply reducing cache memory. It is standard in long-context models.
- Sparse and sliding-window attention. Instead of every position seeing every other, each sees a nearby window or a rule-selected subset — trading a little quality for a lot of length.
- Efficient implementations. Methods like FlashAttention don't change the math, only how memory is read and written, and deliver large measured speedups.
Its Relationship to Transformers and Context Engineering
The Transformer is the design that assembles attention into a complete architecture: attention moves information between positions, the feed-forward layer processes within them, and residuals plus normalization make depth trainable. Attention is the part; the Transformer is the machine.
Context engineering is a direct consequence of attention's constraints. Because the attention budget is finite and more information means a worse signal-to-noise ratio, deciding what goes into each turn's window is a real design problem. The frequently cited observation that models overlook middle-of-context information is describing how attention weight distributes across a long sequence — not the model deliberately skipping, but the relevant signal being diluted by irrelevant material.
Where People Get It Wrong
"Attention weights are the model's explanation." Attention maps look like explanations, but research has repeatedly shown they can't be read as causal ones: the same output can arise from quite different attention distributions, and perturbing the weights doesn't necessarily change the conclusion. Treat it as a useful clue, not a verdict.
"Attention means the model understands." It computes relevance weights — it routes. Relevance often coincides with meaning, but coincidence isn't comprehension, which is part of why hallucination happens.
"A bigger window makes the attention problem go away." A bigger window only pushes the boundary out. The quadratic cost and the dilution of signal across long sequences remain, which is why long-window models still need retrieval, compaction, and division of labor.
How Much You Need to Know
For general users, two sentences suffice: the model works by weighting information according to relevance, so where you put the key information and how explicitly you state it genuinely changes the result; and the longer the input, the more irrelevant material dilutes the key signal, so a tight prompt usually beats a padded one.
If you build things, it's worth understanding the KV cache lifecycle (what invalidates it), GQA's effect on memory, and why passing a "needle in a haystack" long-context test doesn't imply the model handles complex multi-hop reasoning at that length.