A team needs to run sentiment classification over 40,000 archived support tickets and is not latency-sensitive, but wants to minimize per-request cost. According to OpenAI's Batch API documentation, what tradeoff does using the Batch API (instead of the standard synchronous chat completions endpoint) involve?
- The Batch API charges the same per-token price as the synchronous API but guarantees results within 60 seconds regardless of batch size
- The Batch API is free of charge for any volume of requests, but results are only available after a mandatory 7-day waiting period
- The Batch API only accepts a single request per batch, so the team would still need to submit 40,000 separate batch jobs to process all the tickets
- The Batch API offers roughly a 50% cost discount compared to the synchronous API, but processes the submitted batch of requests asynchronously with results typically available within 24 hours rather than immediately, and draws from a separate rate-limit pool
Why D? And why not the others?
Correct answer: D. The Batch API offers roughly a 50% cost discount compared to the synchronous API, but processes the submitted batch of requests asynchronously with results typically available within 24 hours rather than immediately, and draws from a separate rate-limit pool
OpenAI documents the Batch API as accepting a file of many requests, each with its own identifier for matching results back to inputs, that is processed asynchronously against a separate rate-limit pool, at roughly a 50% cost discount relative to the equivalent synchronous requests, with completion typically well within the fixed 24-hour completion window rather than returned immediately -- a tradeoff of latency for cost that fits exactly the non-latency-sensitive bulk classification use case described. The claim of same price but a guaranteed 60-second turnaround is wrong on both counts: the price is discounted relative to the synchronous endpoint, and results are not guaranteed within any such short window. The claim that it is entirely free with a mandatory week-long wait invents figures the documentation does not state; the actual discount is roughly half price, not zero, and the completion window is 24 hours, not 7 days. The claim of a one-request-per-batch limit is backwards: a single batch file is specifically built to bundle a large number of requests, documented up to tens of thousands, into one submission, which is the entire point of the feature for a use case like this one.
Source: OpenAI, Batch API guide, https://developers.openai.com/api/docs/guides/batch