Temperature, top-p, and top-k in LLM APIs: how they combine, with a worked example

Most guides explain temperature, top-k, and top-p one at a time, each with its own toy example, and stop there. That leaves the actual question unanswered: when an API call sets all three at once, what happens, and in what order? This page runs one distribution through the full pipeline with real numbers, so you can check every step by hand instead of taking it on faith.

The pipeline: temperature, then top-k, then top-p

A model doesn't output a token directly. It outputs one raw score (a "logit") per vocabulary entry, and decoding turns those scores into a choice. The standard order, matching how Hugging Face's transformers library builds its list of sampling filters, is:

1. Temperature divides every logit by T before the softmax step, reshaping how peaked or flat the resulting probabilities are.
2. Top-k keeps only the k highest-probability tokens and discards the rest.
3. Top-p (nucleus sampling) then keeps the smallest group of tokens, taken in probability order, whose probabilities add up to at least p.
4. Whatever survives is renormalized to sum to 1, and the next token is sampled from that shortened, rescaled list.

Top-k was introduced by Fan, Lewis and Dauphin (2018), "Hierarchical Neural Story Generation," as a fixed-size cutoff. Top-p was introduced by Holtzman et al. (2019), "The Curious Case of Neural Text Degeneration," specifically to fix top-k's blind spot: a fixed k is too small when the model is genuinely unsure between many tokens, and too large when it's highly confident in one or two.

Step 1: temperature reshapes the distribution

Say the prompt is "The chef added a pinch of ___" and the model's five candidate next-tokens have these raw logits: salt 4.0, pepper 3.0, sugar 2.0, basil 1.0, cinnamon 0.0. Temperature scaling divides each logit by T, then applies softmax. Three settings side by side:

Lower temperature doesn't change the ranking — salt is still most likely at every setting — it changes how lopsided the odds are. At T = 0.5 the model is nearly certain; at T = 1.5 the field is much more open. This is why temperature alone controls "predictable vs. varied" output: it never removes an option, it only redistributes probability mass toward or away from the leader.

Step 2 and 3: why top-k and top-p disagree with each other

The mechanical difference between top-k and top-p only shows up once you compare them on distributions with different amounts of confidence. Take top-k = 3 and top-p = 0.9, and apply both to the sharp (T = 0.5) and flat (T = 1.5) distributions above:

Top-k = 3 keeps exactly three tokens no matter what: on the sharp distribution that's one token more than needed (sugar's 1.6% adds almost nothing), and on the flat distribution it's one token short, silently cutting basil even though basil is carrying real probability mass at that temperature. Top-p adapts its nucleus size to how confident the model actually is at that step — small when the model is sure, larger when it isn't — which is exactly the failure mode Holtzman et al.'s paper set out to fix.

Full worked example: all three parameters in one API call

Now run the whole pipeline on one request: the same five logits, temperature = 0.7, top_p = 0.9, and no top-k limit set (many APIs, including OpenAI's and Anthropic's, expose top_p without requiring top_k).

Cumulative probability crosses the 90% threshold right after pepper (76.1% + 18.2% = 94.3%), so the nucleus is just {salt, pepper} — sugar, basil, and cinnamon are all dropped, even though none of them were literally zero. Renormalizing the surviving pair (94.3% total) gives salt = 76.1 / 94.3 ≈ 80.7% and pepper = 18.2 / 94.3 ≈ 19.3%. The model then samples one token from just those two odds — meaning roughly 4 times out of 5 it produces "salt," and roughly 1 time in 5 it produces "pepper," and it will never produce "sugar," "basil," or "cinnamon" on this particular call.

Why you usually change one, not both

Temperature and top-p both control randomness, but they interact in a way that's hard to predict from either number alone — OpenAI's own API reference notes this directly, recommending you adjust temperature or top_p but not both in the same request. The worked example above shows why: raising temperature widens the distribution first, which then changes how many tokens top-p's cumulative cutoff has to include to reach the same threshold. Push both toward their "more random" ends at once (say temperature = 1.5 with top_p = 0.95) and the nucleus can balloon to include tokens that were barely above noise before scaling, compounding unpredictably rather than adding cleanly. The one common exception worth knowing: pairing a high temperature with a moderate top-p (for example 1.0 with 0.5) as a deliberate safety rail — using top-p to cap how far the temperature is allowed to wander — is a documented, intentional pattern, not an accident.

For questions on how these and other generation parameters are actually exposed across provider APIs — streaming, stop sequences, logprobs, rate limits — see the LLM APIs practice quiz.

Note: sampling behavior for any specific model or endpoint should always be checked against that provider's current API reference before shipping, since exact parameter names, ranges, and defaults do vary and change between providers and model versions.

Source: Fan, Lewis and Dauphin, "Hierarchical Neural Story Generation" (arXiv:1805.04833); Holtzman et al., "The Curious Case of Neural Text Degeneration" (arXiv:1904.09751); Hugging Face transformers library, generation/utils.py (logits warper construction order: temperature, then top-k, then top-p); OpenAI API reference, chat completions parameters (temperature/top_p guidance).

Drill Building with LLM APIs practice questions →