A developer notices that a long-form LLM answer (for example, 'list and explain five approaches to X') is slow to generate because the model must produce the entire response as one long sequential stream of tokens, point by point, in order. Ning et al. (2023), "Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding," propose a prompting-and-decoding approach aimed specifically at cutting this latency without changing the model's weights. What does Skeleton-of-Thought do?
- It first prompts the model to generate a brief skeleton of the answer's main points, then issues separate parallel calls (or batched decoding) to expand each point concurrently, before combining the expanded points into the final answer
- It shortens the answer by instructing the model to omit the elaboration for each point, trading completeness for speed
- It routes the request to a smaller, faster model for a first draft, then upgrades to the original model only for a final proofreading pass
- It caches the model's response to the same prompt across users, so only the first request incurs the full sequential generation cost
Why A? And why not the others?
Correct answer: A. It first prompts the model to generate a brief skeleton of the answer's main points, then issues separate parallel calls (or batched decoding) to expand each point concurrently, before combining the expanded points into the final answer
Skeleton-of-Thought works in two stages: it first prompts the model to produce a short skeleton listing the main points of the eventual answer, and then, instead of continuing to generate the full elaboration sequentially, it issues separate API calls or uses batched decoding to expand each skeleton point at the same time, finally stitching the expanded points together into the complete response. The paper reports considerable speed-ups across a dozen LLMs from this parallelism, and in some cases even better answer quality, because each point is expanded with a focused, self-contained prompt. The option describing omitting elaboration for speed is wrong because Skeleton-of-Thought still generates full elaborated content for every point; it changes when and how that content is generated, not whether it exists. The option describing a smaller draft model handed off to the original model for proofreading is wrong because it describes a draft-then-upgrade or speculative-decoding style pipeline, not the skeleton-then-parallel-expansion structure this paper proposes. The option describing caching a response across users is wrong because that is an infrastructure-level caching strategy operating across separate requests, unrelated to how tokens are decoded within a single answer's generation.
Source: Ning, Lin, Zhou, Yang & Wang, 'Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation' (arXiv:2307.15337, 2023)