In LLM inference APIs such as Anthropic's Messages API and OpenAI's Chat Completions API, what effect does lowering the `temperature` sampling parameter toward 0 have on generated text?
- It sharpens the probability distribution over next tokens so the model more consistently picks the highest-probability token, producing more deterministic, less varied output
- It reduces the model's context window, so fewer previous tokens are considered when generating each new token
- It increases the number of tokens the model is allowed to generate in a single response
- It disables sampling entirely and forces the model to retrieve an exact quote from its training data
Correct answer: A. It sharpens the probability distribution over next tokens so the model more consistently picks the highest-probability token, producing more deterministic, less varied output
Temperature scales the logits before the softmax step that turns them into a probability distribution over next tokens; a temperature near 0 sharpens that distribution so the highest-probability token is chosen almost every time, yielding deterministic, low-variance output, while higher temperatures flatten the distribution and increase randomness and diversity. The second option is wrong because temperature has nothing to do with how much prior context the model attends to -- that is governed by the context window, a separate, unrelated setting. The third option is wrong because output length is controlled by a separate maximum-tokens style parameter, not by temperature. The fourth option is wrong because even at temperature 0 the model is still generating tokens step by step from its learned probability distribution; it is not performing retrieval or reproducing an exact quotation from training data.
Source: Anthropic Messages API reference and OpenAI Chat Completions API reference, `temperature` parameter documentation