The Transformer is a neural network architecture introduced in a 2017 Google paper titled "Attention Is All You Need." The word transform is literal: it takes a sequence of inputs — a passage of text, a strip of image patches, a run of audio frames — and repeatedly transforms it into a representation useful for prediction.
Why it matters is straightforward: nearly every large model you interact with — ChatGPT, Claude, Gemini, Qwen, DeepSeek, and most image and video generators — has a Transformer or a Transformer variant as its skeleton. Understand this one architecture and you understand why models have context windows, why longer inputs cost more, and why they can be trained in parallel at enormous scale.
Grab It in One Sentence First
The Transformer replaced word-by-word sequential processing with an attention mechanism, letting a model read a whole input at once and compute in parallel what each position should pay attention to.
An analogy. An older recurrent network is like one person reading a long contract from beginning to end: by clause 80, the details of clause 3 have blurred, and they have to read one line at a time, so there's no speeding up. A Transformer spreads the whole contract across a conference table, where any clause can look directly at any other and draw a line between them on the spot — and all the clauses can be worked on simultaneously.
Why It Displaced Recurrent Networks
Before 2017, sequence modeling was dominated by RNNs and LSTMs, which pass information forward one time step at a time. That created two hard problems. Long-range dependencies decayed — two distant words struggled to form a connection. And nothing could be parallelized — step t had to wait for step t-1, which capped training scale.
The Transformer removed the sequential constraint. Any two positions in a sequence are one hop apart rather than relayed through everything in between, and a whole input can be processed in a single set of matrix operations. Together, those two properties are what made training models with hundreds of billions of parameters across thousands of GPUs an engineering possibility. They are also the precondition for scaling laws being testable at all — you first need an architecture that can absorb that much compute.
What It's Made Of
Tokenization and embeddings. Text is first split into tokens, and each token is looked up as a vector. Because attention itself has no sense of order, a positional encoding is added to tell the model what came first.
Multi-head self-attention. This is the core, and where the attention mechanism sits in the architecture. Each position computes which other positions it should draw information from and how much of each, then pulls that information in as a weighted sum. "Multi-head" means several such attention patterns run at once — one head may track grammatical structure, another coreference — without interfering.
Feed-forward network. Attention moves information between positions; the feed-forward layer does the processing within each position. Many current large models replace this layer with a mixture of experts, which activates only a fraction of the parameters per token and makes inference cheaper.
Residual connections and normalization. Each layer's output is added back to its input. It looks like a detail, but without it, stacking dozens or hundreds of layers would not train at all.
Encoders, Decoders, and What Won
The original paper's Transformer had two halves: an encoder that read the input and a decoder that wrote the output, for machine translation. Three lineages then split off. Encoder-only models (the BERT family) are good at understanding and classification, and today show up mostly in embedding and retrieval work. Decoder-only models (the GPT family) continue a sequence one token at a time and are overwhelmingly the mainstream for large language models. Encoder-decoder models (the T5 family) remain in use for clean input-to-output tasks like translation and summarization.
Image and video work is converging on the same architecture: slice a picture into patches and treat them as tokens and you have a Vision Transformer; swap a diffusion model's convolutional backbone for a Transformer and you have the DiT design behind much of today's video generation.
Its Cost: Attention Is Quadratic
Standard self-attention computes a relationship between every position and every other position, so doubling the input length roughly quadruples the computation. That is the direct reason context windows cannot grow without limit, and why long documents are noticeably slower and more expensive.
Engineering has a stack of mitigations: efficient implementations like FlashAttention optimize memory traffic, sliding-window and sparse attention restrict what each position can see, and KV caching avoids recomputing what has already been processed — the mechanism that makes prompt caching save money. But the quadratic shape remains, which is exactly why context engineering and context compaction need to exist.
Where People Get It Wrong
"Transformer means large model." It doesn't. The Transformer is only the architecture, the structural design of a building. A large model additionally requires enormous data, a pretraining objective, and alignment tuning. The same architecture at a hundredth of the parameters and data is a completely different thing.
"Attention is the model understanding meaning." Attention computes relevance weights between positions — it is an information-routing mechanism. It often lines up with human intuitions about grammar and reference, but that is a trained statistical result, not evidence that the model knows what it is doing. It is part of why hallucination happens.
"The architecture hasn't changed." The skeleton hasn't; a lot of the detail has. Absolute positional encoding gave way to rotary embeddings (RoPE), normalization moved ahead of the sublayers, attention heads were grouped and shared, and feed-forward layers were replaced by MoE. None of this dethroned the Transformer, but these changes are where recent gains in long context and cheap inference actually came from.
How Much You Need to Know
If you only use AI tools, three facts suffice: models work in tokens, so length is cost; attention is quadratic, so long context gets noticeably slower; and the context window is a hard boundary — anything past it is compressed or dropped.
If you build AI applications, go one step further: understand how KV caching affects time-to-first-token and multi-turn cost, why putting a stable system prompt first lets the cache hit, and why an MoE model's "total parameters" and "active parameters" are not comparable numbers.