LLM temperature is a sampling setting that controls how random a language model’s choice of each next token is. In AI engineering, temperature affects how consistent outputs are from run to run, which matters for evaluation, data extraction, and tool calls.
For example, suppose a model gives three candidate next tokens probabilities of 0.70, 0.20, and 0.10 at temperature 1. At temperature 0.5, those become about 0.907, 0.074, and 0.019. At temperature 2, they become about 0.523, 0.279, and 0.198, so the least likely token is picked about twice as often.
Same model, same prompt. Only the sampling distribution changed.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why temperature matters
Temperature changes randomness in token selection, not the model’s knowledge. It cannot add facts the model does not have. It changes which plausible continuation gets picked.
Temperature matters when you are:
- Running evaluations: Fix temperature so a score change reflects the prompt or model, not sampling luck.
- Extracting structured data: Use a low temperature so the most likely answer is chosen consistently.
- Generating tool arguments: Keep randomness low, because one unlikely token can make a tool call invalid.
- Drafting varied content: Raise temperature moderately when you want several different options.
- Debugging surprising outputs: Check whether a configuration change raised temperature before you rewrite the prompt.
How temperature works
Generation happens one token at a time. The model reads the prompt and the text generated so far, split into tokens. For more on those units, see tokenization.
At each step the model outputs a score, called a logit, for every token in its vocabulary. Sampling turns those logits into probabilities and picks one token.
Temperature is applied at that conversion step. The logits are divided by the temperature T, and the softmax function turns the results into probabilities.
Formula:
p_i = exp(z_i / T) / Σ_j exp(z_j / T)
Where:
- p_i is the sampling probability of token i.
- z_i is the logit for token i.
- Σ_j is the sum over every token j in the vocabulary.
- T is the temperature.
What different temperatures do
T below 1: The distribution sharpens toward the highest-probability token. As T approaches zero, the largest logit dominates. Selection then approaches greedy decoding, which always takes the top token.
T equal to 1: You sample from the model’s own distribution, unchanged.
T above 1: The distribution flattens. Low-probability tokens gain share, so outputs become more varied and errors become more likely.
Framework documentation such as the Hugging Face text generation guide describes temperature alongside top-k and top-p as part of the standard decoding configuration.
How to calculate the effect of temperature
The following Python function applies temperature to a list of logits and returns the new probabilities. It reproduces the three-token example from the opening.
from math import exp, log
def apply_temperature(logits, T):
scaled = [z / T for z in logits]
top = max(scaled)
weights = [exp(z - top) for z in scaled]
total = sum(weights)
return [w / total for w in weights]
# Logits chosen so that T = 1 gives probabilities 0.70, 0.20, 0.10.
logits = [log(0.70), log(0.20), log(0.10)]
for T in (0.5, 1.0, 2.0):
probs = apply_temperature(logits, T)
print(f"T = {T}: " + " ".join(f"{p:.3f}" for p in probs))
Output:
T = 0.5: 0.907 0.074 0.019
T = 1.0: 0.700 0.200 0.100
T = 2.0: 0.523 0.279 0.198
Halving the temperature does not halve the probabilities. Because the transform is exponential, the leading token rises from 0.70 to about 0.91, and the 0.10 token drops to about 0.02.
Doubling the temperature compresses the gaps. The trailing token goes from one pick in ten to about one pick in five.
Temperature vs. top-p and top-k
These are different controls. They are not interchangeable labels for “creativity.”
Temperature: Reshapes the whole probability distribution.
Top-k: Restricts sampling to the k highest-probability tokens.
Top-p: Restricts sampling to the smallest set of tokens whose cumulative probability reaches p. This is also called nucleus sampling.
After a cutoff is applied, the remaining probabilities are renormalized. For more on the cutoff method, see top-p sampling.
Order of operations
Implementations differ on when temperature is applied. Hugging Face Transformers applies the temperature warper before top-k and top-p. That means temperature also changes how many tokens make it into the nucleus.
The Hugging Face generation documentation is a useful reference for how that implementation combines them. Check the order in whatever library or provider you use.
How the controls interact
Lowering top-p while leaving temperature high can still produce erratic picks within the nucleus.
A very low temperature can make generous cutoffs nearly irrelevant, because the top token wins anyway.
Starting points by task
For tasks with one right answer, such as extraction, classification, and tool arguments, start with a low temperature. Leave the cutoffs at provider defaults, then change one control at a time.
For brainstorming or varied drafts, a moderate temperature with top-p below 1 is a common starting point. It keeps very unlikely tokens out while preserving variety among plausible ones.
How to set and test temperature
Hold temperature constant inside any comparison. Suppose prompt variant A ran at 0.8 and variant B ran at 0.2. A quality gap between them may be sampling noise rather than wording.
The experimental discipline in prompt evaluation and LLM evaluation applies unchanged.
Record the sampling settings
Record these settings on every run, so a later reader can tell what was actually compared:
- Temperature: The value used for every call.
- Top-p and top-k: Any cutoffs you set.
- Seed: The random seed, if the provider supports one.
- Model version: The exact pinned model.
The following illustrative config shows the idea. It changes one variable at a time and is recorded with the run.
# Illustrative config: one variable at a time, recorded with the run
config = {"model": "<pinned version>", "temperature": 0.2, "top_p": 1.0}
# Run config against a fixed dataset, multiple samples per case,
# then compare score distributions, not single outputs.
Sample more than once
Because sampling is random, a single generation proves little. Run each configuration several times on the same dataset and compare distributions of scores, not one lucky output.
In production traces, store the sampling parameters on the span. The Phoenix documentation covers capturing generation parameters with traces. Reviewing them alongside outputs is routine AI observability work.
Limitations of temperature
- It cannot fix a bad prompt: At low settings, temperature returns the model’s most likely wrong answer more consistently.
- It does not guarantee determinism: Even at or near zero, providers may not guarantee identical outputs across batching, hardware, or model updates. Exact-match checks should allow for this, or pin everything that can be pinned.
- High values break structure: Very high temperatures quickly degrade coherence and tool-call validity. One unlikely token can derail the tokens that follow, as described for autoregressive models.
- Good values are model-specific: A setting copied from another model or provider needs retesting.
Frequently asked questions
Is temperature 0 fully deterministic?
Not always. It approaches greedy decoding, but providers do not all guarantee identical outputs across requests, batching, or infrastructure changes. Treat it as near-deterministic, and pin the model version and other parameters where reproducibility matters.
Should I use temperature or top-p to control randomness?
Use both deliberately, because they do different jobs. Temperature reshapes probabilities, and top-p limits which tokens are eligible. For single-answer tasks, start with a low temperature. For varied drafts, test a moderate temperature with top-p below 1.
Does higher temperature make a model more creative?
It makes outputs more varied, which can help brainstorming and hurt accuracy. Variety is not the same as quality. Measure it on your task rather than assuming a higher number gives better writing.
Why did my evaluation scores change when I only edited a prompt?
Check whether temperature or other sampling settings differed between the runs. At higher temperatures, sampling noise can swamp a small prompt effect. That is why evaluations fix these settings and sample each case several times.
What temperature should I use for tool calls and JSON output?
Start low. Tool arguments and structured output have one valid shape, and a single unlikely token can make them invalid. Test the setting on your own cases before you rely on it.
Putting temperature to work
Choose a temperature for each task type, and record it with top-p, seed, and model version on every run and trace span.
When you compare prompts or models, keep sampling settings fixed and run several samples per case. Compare score distributions rather than single outputs.
When an output surprises you in production, check the recorded sampling parameters first. Arize Phoenix can show those parameters on each LLM span, next to the output they produced.
