GPT-2 and GPT-3's position-wise feed-forward sublayers use the Gaussian Error Linear Unit (GELU), defined by Hendrycks and Gimpel (2016), instead of the plain ReLU activation used in some earlier architectures. What distinguishes GELU from ReLU?
- GELU weights an input by the value of the standard Gaussian cumulative distribution function evaluated at that input, producing a smooth curve that can pass through small negative values rather than clamping every negative input to exactly zero
- GELU is identical to ReLU for all positive inputs and outputs a small fixed negative constant for every negative input, similar to Leaky ReLU
- GELU replaces the two linear transformations in the feed-forward sublayer with a single linear transformation, removing the nonlinearity between them entirely
- GELU is an activation function that operates only on the attention scores after the softmax, rather than within the position-wise feed-forward sublayer
Why A? And why not the others?
Correct answer: A. GELU weights an input by the value of the standard Gaussian cumulative distribution function evaluated at that input, producing a smooth curve that can pass through small negative values rather than clamping every negative input to exactly zero
Hendrycks and Gimpel define GELU as weighting an input by the value of the standard Gaussian cumulative distribution function evaluated at that same input, which produces a smooth, differentiable curve that, unlike ReLU, does not hard-clamp every negative input to exactly zero -- small negative inputs pass through scaled by a small positive weight rather than being zeroed out entirely. GPT-2 and GPT-3 use this activation between the two linear transformations of their position-wise feed-forward sublayers. The option describing a small fixed negative constant for every negative input is wrong because that describes Leaky ReLU's behavior, a different activation with a fixed negative slope, not GELU's Gaussian-weighted curve. The option describing removal of the nonlinearity is wrong because GELU is itself the nonlinearity inserted between the two linear transformations, not a replacement that eliminates nonlinearity. The option placing this activation after the softmax on attention scores is wrong because GELU is used inside the feed-forward sublayer's two linear transformations, entirely separate from the self-attention computation and its softmax.
Source: Hendrycks & Gimpel, "Gaussian Error Linear Units (GELUs)" (2016), arXiv:1606.08415; used in Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019) and Brown et al., "Language Models are Few-Shot Learners" (GPT-3, 2020)