Decoder-only language models such as GPT are pretrained on a simple self-supervised objective before any instruction tuning or reinforcement learning stage is applied. What is that pretraining objective?
- Predicting whether two randomly paired sentences from the corpus originally appeared next to each other in the source text, using a binary classification loss
- Reconstructing a small number of randomly masked-out tokens scattered throughout an otherwise-visible input sequence, using a masked-token classification loss
- Predicting the next token in a sequence given only the tokens that came before it, using a cross-entropy loss between the model's predicted probability distribution and the actual next token, repeated across every position in the training corpus
- Predicting a scalar reward score for an entire generated sequence, using a regression loss trained on human preference comparisons
Why C? And why not the others?
Correct answer: C. Predicting the next token in a sequence given only the tokens that came before it, using a cross-entropy loss between the model's predicted probability distribution and the actual next token, repeated across every position in the training corpus
GPT-style decoder-only models are pretrained with a next-token prediction objective: at every position in a training sequence, the model produces a probability distribution over the vocabulary conditioned only on the preceding tokens (enforced by causal masking), and a cross-entropy loss compares that predicted distribution against the actual next token in the corpus, averaged across all positions and all training sequences; this simple, self-supervised setup requires no labeled data beyond raw text. The option describing next-sentence-pair classification is wrong because that describes a separate auxiliary objective used by some encoder models, not the autoregressive next-token objective decoder-only GPT-style models are pretrained on. The option describing reconstruction of scattered masked tokens is wrong because that describes the masked-language-modeling objective used by bidirectional encoder models, which is incompatible with the strictly left-to-right causal masking decoder-only models use. The option describing a scalar reward regression trained on preference comparisons is wrong because that describes the separate reward-model training stage used later in an RLHF pipeline, not the initial self-supervised pretraining objective.
Source: Radford et al., "Improving Language Understanding by Generative Pre-Training" (2018), Section 3.1; Radford et al., "Language Models are Unsupervised Multitask Learners" (2019)