Batch Norm vs Layer Norm

Summary:

• Batch Norm -> normalizing each feature across batch.

• Layer Norm -> normalizing the values of each individual example independently.

Deep Dive:

Suppose we have 4 students (examples / rows) and 3 subjects (features), say:

Note: analogous to NN tensor: 4 examples * 3 features

Batch Normalization (feature view):

Analogy: "How did everyone do on the Math test?"

• Batch Norm looks at each feature separately:

  • For math: [2, 1, 18, 5]

  • For DSA: [4, 3, 12, 10]

  • For ML: [3, 5, 24, 22]

How it works:

• It calculates the mean for a specific subject across all students:

  • mean_math

    $$[ \mu_{\text{math}}=\frac{2+1+18+5}{4} ]$$

• var_math:

$$[ \sigma_{\text{math}}^2= \frac{1}{4}\sum_{i=1}^{4}(x_i-\mu_{\text{math}})^2 ]$$

• It repeats this for DSA and ML to get mean_dsa, var_dsa, etc.

Layer Normalization (Row view):

Analogy: "How did Student A perform overall across all their subjects?"

• Layer Norm looks at each example separately:

  • For A -> [2, 4, 3]

  • For B -> [1, 3, 5]

  • For C -> [18, 12, 24]

  • For D -> [5, 10, 22]

How it works

• mean_A:

$$[ \mu_A=\frac{2+4+3}{3} ]$$

• var_A:

$$[ \sigma_A^2= \frac{(2-\mu_A)^2+(4-\mu_A)^2+(3-\mu_A)^2}{3} ]$$

• It repeats this independently for students B, C, and D.

The Engine: Z-Score Normalization

Both norm's used Z-Score normalization only.

$$[ \hat{x} = \frac{x-\mu}{\sqrt{\sigma^2+\epsilon}} ]$$

Note: epsilon is added to avoid divide by zero error. Its a very small value.

When to use what?

Layer Norm:

Use LayerNorm when we want to normalize each example independently, without depending on other examples in the batch.

This is especially useful in sequence models, where each token has its own hidden representation.

Generally used in: Transformers, LLMs, RNNs, LSTMs, GRUs, Vision Models.

Batch Norm:

Use BatchNorm when batch-level statistics are meaningful and useful for training.

Generally used in: CNNs, ResNet-style architectures, Image classification, Object detection, Image segmentation.