LLM generation APIs commonly expose a `temperature` parameter alongside sampling-restriction parameters like `top_p` or `top_k`. Mechanically, what does `temperature` itself do to the model's next-token choice?
- temperature removes the least probable tokens from consideration entirely, functioning identically to `top_k` sampling with the temperature value acting as the value of k
- temperature scales the model's output logits before they are converted into probabilities via softmax; lower values sharpen the resulting distribution toward the highest-probability tokens, while higher values flatten it, making comparatively less probable tokens more likely to be sampled
- temperature is applied only after a token has already been sampled, triggering a re-roll whenever a separate filter model flags the chosen token as low quality
- temperature has no effect on which token is sampled and instead controls only how many separate completions the API returns for a single request
Why B? And why not the others?
Correct answer: B. temperature scales the model's output logits before they are converted into probabilities via softmax; lower values sharpen the resulting distribution toward the highest-probability tokens, while higher values flatten it, making comparatively less probable tokens more likely to be sampled
Temperature is a scaling factor applied to the model's raw output logits before the softmax function converts them into a probability distribution over the vocabulary; dividing logits by a small temperature value sharpens the distribution so the highest-probability tokens dominate even further, approaching a near-deterministic choice as temperature approaches zero, while dividing by a larger value flattens the distribution, giving lower-probability tokens a comparatively higher chance of being sampled. Describing temperature as functionally identical to `top_k` is wrong because `top_k` restricts the candidate pool to a fixed count of the highest-probability tokens rather than reshaping the probability distribution itself, which is a distinct mechanism. Describing temperature as a post-hoc quality filter invents a re-rolling mechanism that generation APIs do not implement as part of this parameter. Describing temperature as controlling the number of returned completions confuses it with an entirely separate parameter, such as a count of choices to generate, that some APIs expose independently.
Source: OpenAI, Chat Completions API reference, `temperature` parameter, https://platform.openai.com/docs/api-reference/chat/create