Some LLM APIs support returning `logprobs` (log probabilities) alongside generated text. What do these values represent, and what are they typically used for?
- They report how many milliseconds the API took to generate each token, and are used purely for latency monitoring and performance debugging
- They report, for each generated token, the log probability the model assigned to it (and often to alternative candidate tokens), and are typically used to gauge the model's confidence or build classifiers from token likelihoods
- They report the total dollar cost billed for each individual token, broken out on a per-token basis for detailed cost accounting
- They report which of several fine-tuned model versions actually generated each token, for use in auditing which checkpoint produced a given response
Why B? And why not the others?
Correct answer: B. They report, for each generated token, the log probability the model assigned to it (and often to alternative candidate tokens), and are typically used to gauge the model's confidence or build classifiers from token likelihoods
Log probabilities express, on a log scale, how likely the model considered each token it actually output (and optionally some number of alternative top candidate tokens at that position), which is useful for gauging the model's confidence in its own output, flagging likely hallucinations where confidence is unusually low, or building lightweight classifiers on top of a generation task by comparing the relative log probabilities of a small set of candidate answers. They have nothing to do with latency, which providers report separately, if at all, as timing metadata rather than as a probability value. They also are not a per-token cost breakdown; billing is typically reported as aggregate input and output token counts rather than a probability-shaped field attached to each token. And they do not identify which fine-tuned checkpoint generated a token -- that information, if available at all, would be reported as separate model or version metadata on the response, not encoded in a probability value.
Source: OpenAI, Chat Completions API reference, `logprobs`/`top_logprobs` parameters, https://platform.openai.com/docs/api-reference/chat/create