Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models" (the Chinchilla paper), trained over 400 models to study how model size and training data should be allocated for a fixed compute budget. What was their central finding?
- For a fixed training compute budget, model parameters and training tokens should be scaled at roughly the same rate, implying substantially larger training datasets relative to model size than many earlier large models had used
- Model performance becomes essentially independent of the number of training tokens once the parameter count exceeds a few billion
- Compute-optimal training requires increasing model size much faster than training data, since parameter count is the dominant driver of capability
- The optimal ratio of training tokens to parameters decreases as total training compute increases, so very large models need proportionally less data
Why A? And why not the others?
Correct answer: A. For a fixed training compute budget, model parameters and training tokens should be scaled at roughly the same rate, implying substantially larger training datasets relative to model size than many earlier large models had used
Hoffmann et al. found that for compute-optimal training, model size and the number of training tokens should be scaled at roughly equal rates -- for every doubling of model size, training tokens should also roughly double, yielding a compute-optimal ratio of around 20 training tokens per parameter. Applying this, their 70-billion-parameter Chinchilla model, trained on about 1.4 trillion tokens, outperformed several much larger contemporary models that had been trained on comparatively fewer tokens relative to their size, showing that many earlier large models were undertrained relative to their parameter count. The option claiming performance becomes independent of token count past a few billion parameters is wrong because the paper's whole point is that additional training tokens keep mattering as models scale. The option claiming model size should scale much faster than data is wrong because it describes the pre-Chinchilla assumption the paper revised, not its finding. The option claiming the optimal token-to-parameter ratio decreases with scale is wrong because Hoffmann et al. found this ratio to be roughly constant across the compute budgets they studied.
Source: Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022), arXiv:2203.15556