Sennrich et al. (2016), "Neural Machine Translation of Rare Words with Subword Units," introduced byte-pair encoding (BPE) as a tokenization method for handling rare and out-of-vocabulary words. How does the BPE algorithm build its subword vocabulary?
- It assigns a fixed-length numeric code to each whole word in a predetermined dictionary, discarding any word not already present in that dictionary
- It randomly samples character sequences of varying length from the training corpus until a target vocabulary size is reached
- It splits text purely along whitespace and punctuation boundaries, performing no further segmentation within a word
- It starts from individual characters and repeatedly merges the most frequent adjacent pair of symbols into a new symbol, learning a fixed number of merge operations to build a subword vocabulary
Why D? And why not the others?
Correct answer: D. It starts from individual characters and repeatedly merges the most frequent adjacent pair of symbols into a new symbol, learning a fixed number of merge operations to build a subword vocabulary
Sennrich et al. describe BPE as starting with each word represented as a sequence of characters plus an end-of-word symbol, with the initial symbol vocabulary consisting of all individual characters. The algorithm then repeatedly counts all adjacent symbol pairs across the training corpus, merges the single most frequent pair into a new symbol, and adds that symbol to the vocabulary; this repeats for a chosen number of merge operations, gradually building larger subword units out of frequent character sequences and even whole common words. This lets rare or unseen words be represented as sequences of known subwords instead of a single out-of-vocabulary token. The option describing whole-word dictionary codes is wrong because that is exactly the closed-vocabulary approach BPE was designed to avoid. The option about random sampling is wrong because merges are chosen deterministically by frequency, not randomly. The option describing pure whitespace/punctuation splitting is wrong because BPE explicitly performs further segmentation within words based on learned merges.
Source: Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units" (2016), arXiv:1508.07909, Section 3.2