When calling a foundation-model API, temperature may be the parameter people fill in most casually and understand least. It is often described as a “creativity knob”: turn it up for more creative output and down for stricter answers. That is not entirely wrong, but it is so imprecise that it encourages misuse. Someone gives a high temperature to a task that requires strict JSON and then complains about unstable formatting; someone else sets brainstorming to 0 and decides the model has “no imagination.”
To use the parameter correctly, you must understand what it actually changes. That begins with one fact: every token a foundation model generates is sampled from a probability distribution. Once you understand that sentence, temperature stops being mysterious.
At every step, the model predicts a probability distribution for the next token. Temperature controls how strongly sampling favors the high-probability candidates.
The Foundation: A Model Outputs Probabilities, Not an Answer
A foundation model generates text one token at a time. At each step, it does not simply “think of the next word.” It assigns a probability to every candidate in its vocabulary. Given a phrase equivalent to “The weather today is really...,” for example, its internal distribution might look like this:
好 → 0.62
不错 → 0.18
冷 → 0.09
糟糕 → 0.05
...（其余成千上万个词，概率很低）It then samples one candidate from that distribution, appends it to the input, predicts another token, and repeats. Temperature changes how that probability distribution is treated during sampling. That is why the parameter belongs to inference: it acts during decoding and does not change the model itself.
What Temperature Does: Flatten or Sharpen the Distribution
Temperature is a scaling parameter applied to the probability distribution. Intuitively:
- Low temperature, approaching 0: The distribution becomes sharper. High-probability tokens receive even more weight, while low-probability options approach zero. At the extreme, temperature=0 selects the most likely token at each step—greedy decoding—so output becomes nearly deterministic and the same input produces almost the same response each time.
- High temperature, such as above 1.0: The distribution becomes flatter, giving originally unlikely tokens more opportunity to be selected. The output becomes more varied and divergent, but also more likely to wander, become awkward, or go off topic.
Return to the example above. At low temperature, the model will almost always select the most probable completion; at high temperature, words corresponding to “cold” or “terrible” gain a meaningful chance, creating a more surprising continuation. What we call creativity is essentially allowing the model to choose words it considers less likely but still plausible.
This also explains a common puzzle: why is temperature=0 not always 100% stable? In real systems, floating-point variation during parallel computation and tie-breaking between equally likely tokens can still introduce tiny differences. Strict reproducibility also requires a fixed random seed and other controlled conditions, and implementations differ across vendors.
Top-p and Top-k: Other Controls Used with Temperature
Temperature often appears alongside top_p, or nucleus sampling. They control randomness from different directions:
- Top-k: Sample only from the k highest-probability tokens and discard every other candidate;
- Top-p, or nucleus sampling: Sample from the smallest set of tokens whose cumulative probability reaches p. With top_p=0.9, candidates are sorted from most to least likely and added until their cumulative probability reaches 90%; sampling then occurs within that set.
These parameters define the candidate set first, while temperature determines preferences inside that set. A common practical recommendation is to focus on adjusting only one—temperature alone is sufficient in most situations. Pushing both to extremes makes their interaction difficult to predict. See the model parameters entry for more definitions.
Top-p defines the candidate set first; temperature then determines preferences within it. The two control randomness from different directions.
Recommended Settings for Different Tasks
There is no universal temperature, only a temperature appropriate to the task. The following is a useful starting point; reasonable ranges vary by model, so validate them in practice:
| Task type | Recommended direction | Reason |
|---|---|---|
| Structured output, JSON, code, and extraction | Very low, near 0 | The result must be stable, parseable, and reproducible; variety adds no value |
| Factual Q&A, translation, and summarization | Low | Accuracy matters, so reduce room for improvisation |
| Classification, decisions, and scoring | Very low | Results should be consistent for identical inputs |
| Everyday conversation and customer support | Medium | Natural without losing control |
| Copywriting, naming, and brainstorming | Moderately high | Diversity and surprise are useful |
| Creative writing and poetry | High | Encourage divergence |
A frequent mistake is using a relatively high temperature for a deterministic task. A JSON object occasionally loses a bracket, classifications fluctuate, or an agent’s tool-calling parameters become unstable. Many complaints that “the model is unreliable” begin here. Conversely, people who find model output repetitive and uncreative have often set temperature too low.
A Reproducible Comparison Experiment
Build intuition by running a controlled comparison rather than memorizing a parameter table:
- Fix one prompt, such as “Give me five names for a budgeting app focused on privacy”;
- Change only temperature, running three trials each at 0, 0.7, and 1.2;
- Observe two things: how much the three runs differ at the same temperature, which measures randomness, and how style changes across temperatures, which shows divergence.
The pattern will be obvious. At temperature=0, the three runs are almost identical and the names conservative. At 1.2, every run differs; some ideas are inspired and others absurd. Repeat the experiment with “convert this paragraph to JSON” and JSON errors will begin to appear at high temperature. You will then understand why structured tasks need low temperature instead of merely memorizing the advice. This kind of minimal, experiment-driven understanding is fundamental to evaluation and parameter tuning.
Control the variables, change one parameter, and run several trials. A ten-minute comparison teaches more than repeatedly reading parameter documentation.
Who This Is For and What to Keep in Mind
This guide is for developers and prompt engineers who call foundation-model APIs directly and need to tune parameters. Keep the following points in mind:
- [Reasoning models](/wiki/reasoning-model) may behave differently: Some reasoning-focused models respond to temperature differently from conventional models, and their vendors may recommend leaving the default unchanged. Read the model documentation before tuning;
- Temperature does not solve capability limitations: If a model does not know something, raising temperature only makes it invent more confidently. The parameter changes how words are selected, not what the model knows;
- Fix temperature during evaluation: In model comparisons or prompt A/B tests, temperature must remain constant or you cannot distinguish model differences from random variation.
Frequently Asked Questions
Q: Does temperature=0 guarantee identical output? A: It is very close to deterministic, but as explained above, implementation details can still produce tiny variations. Strict reproducibility requires additional controls such as a fixed seed and depends on platform support. A value of 0 is enough for broadly stable output; byte-for-byte reproducibility requires checking the specific provider.
Q: Should I adjust temperature or top_p? A: In most cases, adjust temperature and leave top_p at its default. Changing both makes behavior harder to predict. For finer control, another common approach is to fix temperature=1 and narrow the candidate set only with top_p. Choose one style and use it consistently.
Q: Why does a web-based chat assistant answer the same question differently each time? A: These products usually use a nonzero temperature to make conversation feel more natural. That is intentional, not a bug. Lower the temperature in an API call to obtain more stable answers.
Conclusion
Temperature is not a mystical creativity knob; it controls randomness during sampling. A lower value sharpens the distribution and makes output more deterministic, while a higher value flattens it and makes output more divergent. The key is remembering that a model produces a probability distribution at every step. Usage follows naturally: lower it for stable, parseable tasks; raise it when diversity and surprise matter; and hold it constant during evaluation. To make the idea stick, spend ten minutes on a controlled comparison instead of memorizing another parameter table.