An LLM generation API exposes both a `top_p` (nucleus sampling) parameter and, in some APIs, a `top_k` parameter, in addition to `temperature`. How does nucleus sampling with `top_p` differ from `top_k` sampling in how it restricts the model's next-token choices?
- `top_p` and `top_k` are two names for the exact same mechanism, differing only in whether the cutoff value is expressed as a percentage or as a raw integer
- `top_k` restricts choices based on cumulative probability mass, so its cutoff point moves depending on how confident the model is at each step, while `top_p` always keeps exactly the same fixed number of candidates
- `top_p` restricts sampling to the smallest set of most-probable tokens whose cumulative probability reaches the threshold `p`, so the number of candidates varies step to step, while `top_k` always keeps a fixed number of the highest-probability tokens regardless of how the probability mass is distributed
- Both parameters only take effect when `temperature` is set to exactly 0, and have no effect on generation at any other temperature value
Why C? And why not the others?
Correct answer: C. `top_p` restricts sampling to the smallest set of most-probable tokens whose cumulative probability reaches the threshold `p`, so the number of candidates varies step to step, while `top_k` always keeps a fixed number of the highest-probability tokens regardless of how the probability mass is distributed
Nucleus sampling (`top_p`) builds a candidate pool by adding tokens in order of probability until their cumulative probability reaches the threshold `p`, so a peaked distribution yields a small pool and a flatter distribution yields a larger one -- the pool size adapts to the model's confidence at each generation step. `top_k` sampling instead always keeps exactly the same fixed number, `k`, of the highest-probability tokens as candidates regardless of how concentrated or spread out the probability mass is, which can include unlikely tokens when the true distribution is sharply peaked, or exclude reasonable ones when it is flat. The two are not the same mechanism expressed in different units, since one produces a variable-size candidate pool and the other a fixed-size one. The option describing `top_k` as the variable, confidence-dependent one and `top_p` as fixed reverses their actual behavior. Both parameters shape the candidate pool independently of `temperature` and take effect at any temperature setting, not only at exactly 0, where sampling becomes fully deterministic and such cutoffs become moot only in the trivial sense that a single token already dominates the distribution.
Source: OpenAI, API reference `top_p` parameter documentation, https://platform.openai.com/docs/api-reference/chat/create; Holtzman et al. (2020), 'The Curious Case of Neural Text Degeneration' (arXiv:1904.09751), introducing nucleus sampling