A developer needs an LLM to classify 600 short product reviews as positive or negative, paying per input-and-output token through a hosted API. Sending each review as its own separate call means the same lengthy classification instructions and few-shot examples are repeated in every single request. Cheng et al. (2023), "Batch Prompting: Efficient Inference with Large Language Model APIs," propose an alternative. What does batch prompting do to cut this cost?
- It fine-tunes a smaller specialized model on the same classification examples so the large model is no longer needed once training completes
- It groups several independent input samples together into a single prompt, sharing one copy of the instructions and in-context examples across the whole group, and has the model return the answers for all of the grouped samples in one response instead of issuing a separate call per sample
- It reduces cost by having the model skip the few-shot examples entirely and rely purely on zero-shot instructions for every request
- It caches every previous request-response pair, so a later request only costs tokens if its text is not an exact character-for-character match to an earlier one
Why B? And why not the others?
Correct answer: B. It groups several independent input samples together into a single prompt, sharing one copy of the instructions and in-context examples across the whole group, and has the model return the answers for all of the grouped samples in one response instead of issuing a separate call per sample
Batch prompting groups multiple independent input samples into a single prompt so that the shared instructions and few-shot demonstrations only need to appear once per batch rather than once per sample, and it asks the model to return the answers for every grouped sample together in one response; because the fixed overhead of instructions and examples is amortized across several samples, the paper reports the token and time cost of inference dropping nearly linearly with the number of samples placed in each batch, while accuracy stays comparable to the one-sample-per-call approach. The option describing fine-tuning a smaller model is wrong because batch prompting is a pure prompting technique applied to the existing model through its ordinary API, with no separate training step. The option describing dropping the few-shot examples is wrong because batch prompting keeps the same examples; it just stops repeating them once per sample. The option describing caching exact-match requests is wrong because batch prompting changes how many samples are packed into a single call, not whether previously seen text is reused from a cache.
Source: Cheng, Kasai & Yu, 'Batch Prompting: Efficient Inference with Large Language Model APIs' (arXiv:2301.08721, 2023)