Ba, Kiros and Hinton (2016), "Layer Normalization," introduced a normalization technique widely used in transformer blocks. How does layer normalization compute its statistics, and how does this differ from batch normalization?
- It normalizes each feature by computing statistics across every example in the mini-batch, exactly as batch normalization does, but applies the result at a different point in the network
- It normalizes activations using running statistics collected only during a separate calibration pass performed after training has finished
- It computes the mean and variance across the feature dimension for each individual training example, making it independent of batch size, whereas batch normalization computes statistics across the batch for each feature
- It normalizes only the query and key projections used in self-attention, leaving the value projection and feed-forward outputs unnormalized
Why C? And why not the others?
Correct answer: C. It computes the mean and variance across the feature dimension for each individual training example, making it independent of batch size, whereas batch normalization computes statistics across the batch for each feature
Ba et al. define layer normalization as computing the mean and variance across the feature dimension for a single training example, then using those statistics to normalize that example's activations; because the computation only ever looks within one example, it does not depend on how many examples are in a mini-batch. This differs from batch normalization, which instead computes the mean and variance for each feature across all examples in the current mini-batch. This batch-size independence is why layer normalization suits transformers and recurrent networks, where batch statistics can be unstable or awkward to define. The option describing batch-wide statistics is wrong because that describes batch normalization, not layer normalization. The option describing a post-training calibration pass is wrong because layer normalization's statistics are computed on the fly for every forward pass, during both training and inference. The option restricting normalization to only the query and key projections is wrong because layer normalization is applied to entire sublayer outputs, not to specific attention projections.
Source: Ba, Kiros & Hinton, "Layer Normalization" (2016), arXiv:1607.06450, Section 3