Top-p sampling is a text generation method that picks each next token only from the most likely tokens whose probabilities add up to p. It is one of the main settings that controls how predictable or varied a model’s output is, so it directly affects evaluation results and production behavior.
For example, suppose the model’s top five candidates for the next token have probabilities 0.50, 0.25, 0.15, 0.06, and 0.04. With top-p set to 0.90, only the first three tokens are kept, because 0.50 + 0.25 + 0.15 = 0.90. The model then samples one of those three, and the 0.06 and 0.04 tokens cannot be chosen at this step.
Top-p sampling is also called nucleus sampling. The kept set of tokens is the “nucleus.”
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why top-p sampling matters for AI engineering
A large language model (LLM) does not produce one fixed next token. It assigns a probability to every token in its vocabulary, and a decoding method decides which one to use.
Many of those tokens are individually unlikely. Sampling one of them, even once, can derail everything generated after it. Top-p cuts off that unreliable tail.
Understanding top-p helps with:
- Tuning output style: Make generation more conservative for extraction or more varied for drafting.
- Running fair comparisons: Hold decoding settings constant so a prompt change, not sampling noise, explains the difference.
- Debugging odd outputs: Check the recorded decoding settings before you rewrite a prompt.
- Reading API settings: Understand how top-p differs from top-k and temperature, which providers often expose side by side.
How top-p sampling works
At each generation step, top-p follows the same procedure.
- Score every token: The model assigns a probability to every token in its vocabulary.
- Sort: The tokens are sorted from most to least probable.
- Accumulate: Walk down the sorted list, adding up probabilities until the running total reaches p.
- Cut: Every token past that point is excluded for this step.
- Renormalize: The kept probabilities are scaled so they add up to 1.
- Sample: One token is drawn at random from the kept set.
For more on the units the model is choosing between, see tokenization.
The cutoff is probability mass, not a token count
Top-p does not keep a fixed number of tokens. It keeps however many tokens it takes to reach p.
When the model is confident, one or two tokens may already add up to p, so the nucleus is small. When the model is uncertain and the probabilities are spread out, the nucleus grows.
Suppose the model is torn between two continuations at 0.45 and 0.40. Together they add up to 0.85, which is below 0.90. So the p = 0.90 nucleus holds those two tokens plus the next token that pushes the total past 0.90.
This adaptivity is the main reason to use top-p. The candidate set tracks the model’s uncertainty, step by step.
How to calculate a top-p nucleus
The following script applies top-p to the distribution from the opening example. It shows the kept tokens and their renormalized probabilities at p = 0.90 and p = 0.75.
It also applies a temperature of 0.5 first, then top-p at 0.90, to show how the two settings interact. Raising each probability to the power 1/T and renormalizing gives the same result as dividing the logits by T before the softmax.
def nucleus(probs, p):
ranked = sorted(probs.items(), key=lambda item: -item[1])
kept, total = [], 0.0
for token, prob in ranked:
kept.append((token, prob))
total += prob
if total >= p - 1e-9:
break
return {token: prob / total for token, prob in kept}
def apply_temperature(probs, t):
scaled = {token: prob ** (1 / t) for token, prob in probs.items()}
total = sum(scaled.values())
return {token: value / total for token, value in scaled.items()}
probs = {"A": 0.50, "B": 0.25, "C": 0.15, "D": 0.06, "E": 0.04}
for p in (0.90, 0.75):
kept = nucleus(probs, p)
print(f"top-p {p:.2f}: " + ", ".join(f"{t}={v:.3f}" for t, v in kept.items()))
cooled = apply_temperature(probs, 0.5)
print("T=0.5 probs: " + ", ".join(f"{t}={v:.3f}" for t, v in cooled.items()))
kept = nucleus(cooled, 0.90)
print("T=0.5, top-p 0.90: " + ", ".join(f"{t}={v:.3f}" for t, v in kept.items()))
Output:
top-p 0.90: A=0.556, B=0.278, C=0.167
top-p 0.75: A=0.667, B=0.333
T=0.5 probs: A=0.735, B=0.184, C=0.066, D=0.011, E=0.005
T=0.5, top-p 0.90: A=0.800, B=0.200
The small tolerance in nucleus guards against floating-point rounding when the running total lands exactly on p.
Worked example: one distribution, three settings
Here is the same next-token distribution under different settings:
| Setting | Tokens kept | Renormalized probabilities |
|---|---|---|
| top-p 0.90 | A, B, C | 0.556, 0.278, 0.167 |
| top-p 0.75 | A, B | 0.667, 0.333 |
| temperature 0.5, then top-p 0.90 | A, B | 0.800, 0.200 |
At top-p 0.90, token D at 0.06 was plausible enough to appear in a top-five list. It still falls outside the 90% nucleus and cannot be sampled.
Lowering top-p to 0.75 drops token C as well. Generation becomes more conservative.
Applying a temperature of 0.5 first sharpens the distribution, so A rises to 0.735. Now only two tokens are needed to reach 0.90, and token C drops out even though top-p did not change.
Top-p vs. top-k vs. temperature
These three settings are often confused, but each does a different job.
Top-k keeps a fixed count: the k most probable tokens, whatever their probabilities. On a confident step, top-k = 10 can admit tokens with near-zero probability just to fill the quota. On an uncertain step, it may cut off a genuinely competitive eleventh token.
Top-p adapts the count to the probability mass. That is why top-p and top-k are not interchangeable.
Temperature reshapes the probabilities themselves. It divides the logits by T before the softmax, which sharpens or flattens the distribution that top-p then cuts. For more, see LLM temperature.
A practical way to think about it: top-p and top-k decide which tokens are eligible, and temperature decides how adventurous sampling is among them.
Order matters
In Hugging Face Transformers, for example, temperature is applied before top-p filtering. So temperature changes which tokens make it into the nucleus, as the worked example shows.
A low temperature can make a generous top-p nearly irrelevant, because the top token dominates the nucleus anyway. The order and details vary by stack, so check your provider’s or framework’s documentation.
The Hugging Face text generation documentation describes nucleus sampling alongside the other standard decoding controls.
How to choose and test a top-p value
Start from the task.
- Single-answer tasks: For extraction, classification, and structured tool arguments, start with low randomness overall. Use a low temperature and leave top-p at or near the provider default, then adjust one setting at a time.
- Open-ended drafting: A moderate temperature with top-p around 0.9 is a common starting point. Treat it as a hypothesis to test, not a recommendation to adopt.
Whatever you choose, keep it fixed across compared runs and store it on the trace. Sampling variation at permissive settings can be larger than the effect of the prompt change you are testing. Prompt evaluation and LLM evaluation both depend on holding decoding settings constant.
Sample repeatedly when you evaluate. One generation at top-p 0.95 tells you almost nothing about that configuration. A distribution of scores from repeated samples on a fixed set tells you much more.
When a production output looks wrong, check the recorded decoding settings on the span before rewriting the prompt. Reviewing generation settings in traces is routine AI observability work, and Phoenix captures them alongside outputs.
Limitations of top-p sampling
- It cannot fix a bad distribution: If the model’s top candidates are all wrong, the nucleus samples among wrong answers.
- A low p can drop the right token: On an uncertain step, the correct token may rank just below the cutoff and be excluded.
- Settings do not transfer: The same p behaves differently on a peaked distribution than on a flat one, and a value tuned on one model may misbehave on another.
- Early errors still compound: Generation proceeds one token at a time, so one unlucky draw early affects everything after it. Top-p reduces that risk but does not remove it.
Frequently asked questions
Is top-p the same as nucleus sampling?
Yes. Nucleus sampling is the original name for the method, and top-p is the parameter name most APIs use. Both refer to the same cumulative-probability cutoff.
Should I tune top-p or temperature first?
Change one at a time on a fixed evaluation set. For single-answer tasks, lowering temperature usually has the larger and more predictable effect, while top-p guards against tail tokens. Record both settings either way.
What does top-p = 1 mean?
Top-p = 1 means no nucleus restriction, so sampling uses the full distribution, still shaped by temperature. In most implementations, top-p = 0 or very small values collapse toward always picking the most probable token.
Why do two runs with the same settings give different outputs?
Sampling is random by design, and providers may not offer full determinism even with a seed. Compare configurations over repeated samples and score distributions, not single generations.
Putting top-p sampling to work
Record top-p, top-k, and temperature with every experiment and every production trace. Without them, you cannot reproduce or compare results.
Pick one starting configuration per task type, then change one setting at a time on a fixed evaluation set. Run several samples per case and compare score distributions.
When an output looks wrong, check the decoding settings on the trace first. Tools such as Arize Phoenix record those settings next to each output, so you can tell a sampling problem from a prompt problem.
