What does the `stop` (or "stop sequences") parameter available in most LLM generation APIs do?
- It tells the API to stop generating further tokens as soon as any one of a specified list of strings appears in the output, and to exclude that string from the returned text
- It sets a maximum wall-clock time limit in seconds after which the API forcibly terminates the connection regardless of how much text has been generated
- It specifies a list of words the model is never permitted to generate anywhere in its response, causing an error if any of them would otherwise be produced
- It pauses generation partway through and waits for the client to send an approval signal before continuing to generate the remainder of the response
Why A? And why not the others?
Correct answer: A. It tells the API to stop generating further tokens as soon as any one of a specified list of strings appears in the output, and to exclude that string from the returned text
Stop sequences let a caller specify one or more strings that, if generated, cause the API to immediately halt generation at that point and return the output produced so far, with the matched stop string itself typically omitted from the returned text; this is commonly used to cut a response off at a natural boundary, such as before a model would start role-playing the next turn of a conversation. A wall-clock timeout is a distinct, separate control unrelated to matching specific text content, and would cut a response off regardless of what was actually generated. A blanket prohibition on ever producing certain words describes content filtering or constrained decoding, not what stop sequences do, since stop sequences only trigger termination once a match completes rather than preventing the string from ever being produced. Pausing for manual approval mid-stream describes an entirely different, human-in-the-loop interaction pattern that generation APIs do not implement via this parameter.
Source: OpenAI, Chat Completions API reference, `stop` parameter, https://platform.openai.com/docs/api-reference/chat/create; Anthropic, Messages API reference, `stop_sequences` parameter, https://platform.claude.com/docs/en/api/messages