Radford et al. (2019), "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper), tokenize text using a byte-level variant of byte-pair encoding rather than applying BPE directly to Unicode characters. What problem does operating on bytes instead of Unicode characters solve?
- It allows the tokenizer to skip the merge-based vocabulary-building process entirely, since every byte value can be mapped directly to a token without any merges
- It increases the base vocabulary size well beyond 100,000 symbols before any merges are added, giving the model more starting granularity to work with
- It keeps the base vocabulary, before any merges, small and fixed at 256 symbols (one for every possible byte value), avoiding the far larger base vocabulary that applying BPE directly to Unicode characters would require, and letting the model assign a probability to any input without ever needing an unknown-token symbol
- It removes the need for any subword merging at all, since GPT-2 processes each byte independently as its own token throughout generation
Why C? And why not the others?
Correct answer: C. It keeps the base vocabulary, before any merges, small and fixed at 256 symbols (one for every possible byte value), avoiding the far larger base vocabulary that applying BPE directly to Unicode characters would require, and letting the model assign a probability to any input without ever needing an unknown-token symbol
Radford et al. note that applying BPE directly at the Unicode character level would start from a base vocabulary of over 130,000 symbols before any merges are learned, far larger than the 32,000-to-64,000-token vocabularies typically used; operating on bytes instead means the base vocabulary before any merges is fixed at exactly 256 symbols, one for every possible byte value, and the byte-pair merge process is then run on top of that small, fixed base, eventually growing GPT-2's vocabulary to 50,257 tokens. Because every possible input can be represented as some sequence of bytes, the model can assign a probability to any string at all, removing the need for an unknown-token fallback. The option describing skipping the merge process entirely is wrong because GPT-2 still runs the same iterative BPE merge algorithm on top of the byte-level base vocabulary; only the starting point changes. The option claiming the base vocabulary grows beyond 100,000 symbols is wrong because the entire point of operating on bytes is to keep that base vocabulary small, fixed at 256, not to expand it. The option claiming each byte remains its own separate token throughout generation is wrong because the merge operations combine frequently co-occurring byte sequences into larger subword tokens, exactly as in ordinary BPE.
Source: Radford et al., "Language Models are Unsupervised Multitask Learners" (2019), Section 2.2 (Input Representation)