Shazeer et al. (2017), "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer," introduced a technique now used in some large language models to scale model capacity without a proportional increase in per-token compute. How does a sparsely-gated mixture-of-experts (MoE) layer achieve this?
- A gating network activates every expert sub-network for every input and averages their outputs, providing an ensembling effect at the cost of proportionally higher compute
- A trainable gating network routes each input to a sparse subset of expert feed-forward sub-networks, letting total parameter count scale far beyond what would be affordable if every expert were computed for every input
- Each expert is a full independent copy of the entire transformer stack, and the gating network selects which single copy runs the whole forward pass for a given input
- The gating network is fixed at random initialization and never trained, relying purely on random routing to balance load across experts
Why B? And why not the others?
Correct answer: B. A trainable gating network routes each input to a sparse subset of expert feed-forward sub-networks, letting total parameter count scale far beyond what would be affordable if every expert were computed for every input
Shazeer et al. introduce a sparsely-gated mixture-of-experts layer consisting of up to thousands of feed-forward expert sub-networks, paired with a trainable gating network that, for each input, selects and combines only a small sparse subset of those experts rather than running all of them. Because only the selected experts do any computation for a given input, the total number of parameters in the layer can grow enormously -- the paper reports capacity increases of over 1000x -- while the compute cost per input stays close to that of a much smaller dense network. The option describing activation of every expert with averaged outputs is wrong because that describes a dense ensemble, which is exactly the proportional-compute-cost approach this sparse gating avoids. The option describing experts as full copies of the entire transformer stack is wrong because the experts in this layer are feed-forward sub-networks, not complete duplicated models. The option describing an untrained, randomly fixed gate is wrong because the gating network's parameters are learned jointly with the rest of the model.
Source: Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (2017), arXiv:1701.06538