An LLM API's "context window" limit of, say, 200,000 tokens applies to what, specifically, during a single API call?
- The combined total of the input tokens sent in the request (system prompt, conversation history, and any injected documents) plus the output tokens the model generates in response
- Only the tokens in the user's most recent message, since earlier turns in the conversation are automatically summarized and don't count against the limit
- Only the tokens the model generates in its response, since input tokens are processed by a separate, effectively unlimited ingestion pipeline
- The number of separate API requests a client can make per minute before being rate-limited
Why A? And why not the others?
Correct answer: A. The combined total of the input tokens sent in the request (system prompt, conversation history, and any injected documents) plus the output tokens the model generates in response
The context window is a hard ceiling on the total number of tokens the model can attend to in one call, and that total includes everything fed in as input (system prompt, prior conversation turns, retrieved documents, tool definitions) plus every token the model produces as output; if the combined total would exceed the limit, the request is rejected or there is no room left for the model to generate. Treating only the latest message as counted is wrong because unless a client deliberately truncates or summarizes prior turns itself before sending the request, full conversation history sent as input still counts toward the same shared budget -- there is no automatic summarization performed by the API. Treating only output tokens as counted is wrong because input processing consumes the same limited context, not a separate unlimited channel. Confusing this with a per-minute request cap conflates the context window with rate limiting, a completely separate constraint governing how many calls can be made over time, not how much content fits within one call.
Source: Anthropic, 'Models overview' context window documentation, https://platform.claude.com/docs/en/about-claude/models/overview; OpenAI, 'Models' documentation on context window, https://platform.openai.com/docs/models