Chain of thought, usually abbreviated CoT, means having a model write out its intermediate reasoning step by step before producing a final answer, rather than jumping straight to a conclusion.
The practice was systematized in a 2022 paper with a blunt finding: on arithmetic, commonsense, and symbolic reasoning tasks, simply prompting the model to think step by step raised accuracy substantially — and the larger the model, the larger the gain. The collapsed "thinking…" section you see in today's reasoning models is chain of thought turned into a product feature.
Grab It in One Sentence First
Chain of thought is making the model show its scratch paper instead of only handing in the answer.
Think of a school maths exam where you're required to show your working. It isn't only about partial credit — writing the steps forces you to break the problem apart and cuts down on leaping to a wrong conclusion. Models are in a very similar position: they generate one token at a time, each one building on what's already written. Letting them write the steps first gives them more room to compute and to correct course.
Why It Works
One direct explanation: the computation a model can do in a single forward pass is bounded, and spreading reasoning across dozens or hundreds of tokens converts a one-shot judgment into an iterative one. Once intermediate steps are written out, they're in the context, so later generation can refer back to them — effectively a working memory.
The original paper's method is few-shot CoT: include a few examples of "question + full reasoning + answer" in the prompt, and the model follows the format with its own reasoning. It was later found that appending a phrase like "let's think step by step" often triggers a similar effect on its own, known as zero-shot CoT.
From Prompting Trick to Training Objective
The real shift came when chain of thought moved into training. Reasoning models no longer depend on you asking them to think. Reinforcement learning trains them to produce an extended reasoning pass by default, and to check themselves, back up, and retry along a different path.
That turned chain of thought from a prompting trick into a form of test-time compute: the longer it thinks and the more reasoning tokens it produces, the better it does on hard problems. The cost is equally direct — higher latency and higher bills, because reasoning tokens are billed too.
How It Differs From Neighboring Ideas
Versus prompt engineering. Chain of thought is the single most effective pattern in prompt engineering, but only one of many. On today's reasoning models, manually adding "think step by step" yields little and can even hurt — the model is already doing it internally, and repeating the instruction may disturb its formatting.
Versus an agent. Chain of thought happens entirely inside the model and produces text. An agent turns intermediate steps into real actions — calling tools, reading files, issuing requests — and reasons on from actual results. CoT is thinking; an agent thinks, acts, then thinks about what came back.
Versus context engineering. Long reasoning consumes a lot of window budget. Keeping every turn's full chain of thought across a multi-turn conversation fills the window fast, so in practice only conclusions are retained — a context engineering problem whose mechanism is context compaction.
Where People Get It Wrong
"The written reasoning is the model's real thought process." This is the one to be careful about. Research shows a model's written reasoning does not always match the internal computation it actually relies on: it may form a leaning first and then produce plausible-looking justification, or use a cue in its reasoning without stating it. Chain of thought is highly readable, but it isn't a trustworthy self-report and certainly isn't audit evidence.
"Chain of thought always helps." On simple tasks, forcing long reasoning introduces avoidable errors while adding cost and latency. Gains concentrate on tasks with genuine multi-step structure and explicit intermediate quantities.
"Chain of thought eliminates hallucination." It doesn't. The reasoning itself can confidently invent intermediate facts and then derive a wrong conclusion from them, which can make the error more persuasive. Suppressing hallucination takes retrieval, guardrails, and evals.
Using It Economically
With ordinary chat models, explicitly asking for steps is usually worth it on multi-step calculation, logical judgment, and conditional filtering. With reasoning models, invert that: put your effort into stating the problem and constraints clearly, and leave reasoning depth to the model's own effort setting.
Where cost matters, enable long reasoning only on hard cases: answer cheaply first, then escalate to a reasoning model when the item looks difficult or confidence is low. That's one of the standard strategies in model routing.