The massive performance gap between contiguous DS and node-base structures is due to CPU Cache lines and spatial locality.

Deep Dive:

RAM Bottleneck:

CPU is very fast, but RAM is slow. To prevent CPU from idling, CPU is fitted with CPU Caches (L1, L2, L3).

This includes 2 major things:

• Spatial Locality: States that, if a program accesses a specific memory address, it will almost certainly need the data at the immediately adjacent address very soon.

• Cache Lines: Because of spatial locality, CPU grabs a fixed-size block of contiguous memory all at once. Generally it's 64 bytes long contiguous block in modern machines.

Contiguous Memory DS (eg. Arrays):

Perfect spatial locality.

When you read the first item (say, 4-byte integer) in an array, the CPU fetches a 64-byte cache line from RAM. This brings first integer + the next 15 integer immediately. When the loop moves to the 2nd, 3rd, ..... items, the CPU doesn't have to wait for RAM; the data is already present in the L1-cache (a cache hit). You only suffer a slow RAM fetch once every 16 steps.

Non-contiguous Memory DS (eg. Linked List):

Linked List nodes are dynamically allocated, meaning they are scattered randomly across the heap wherever there happens to be free space.

When you ask for the first node, the CPU fetches a 64-byte cache line from RAM. However, because the next node is stored at a completely different, random memory address, the rest of that cache line is filled with unrelated data.

When we follow pointer to 2nd node, the CPU discovers it isn't in the cache (a cache miss). It has to make a slow trip to RAM again. We have to pay the massive latency penalty on almost every step, making linked list traversal slower regardless of theoretical speed.

• Batch Norm -> normalizing each feature across batch.

• Layer Norm -> normalizing the values of each individual example independently.

Deep Dive:

Suppose we have 4 students (examples / rows) and 3 subjects (features), say:

Note: analogous to NN tensor: 4 examples * 3 features

Batch Normalization (feature view):

Analogy: "How did everyone do on the Math test?"

• Batch Norm looks at each feature separately:

  • For math: [2, 1, 18, 5]

  • For DSA: [4, 3, 12, 10]

  • For ML: [3, 5, 24, 22]

How it works:

• It calculates the mean for a specific subject across all students:

  • mean_math

    $$[ \mu_{\text{math}}=\frac{2+1+18+5}{4} ]$$

• var_math:

$$[ \sigma_{\text{math}}^2= \frac{1}{4}\sum_{i=1}^{4}(x_i-\mu_{\text{math}})^2 ]$$

• It repeats this for DSA and ML to get mean_dsa, var_dsa, etc.

Layer Normalization (Row view):

Analogy: "How did Student A perform overall across all their subjects?"

• Layer Norm looks at each example separately:

  • For A -> [2, 4, 3]

  • For B -> [1, 3, 5]

  • For C -> [18, 12, 24]

  • For D -> [5, 10, 22]

How it works

• mean_A:

$$[ \mu_A=\frac{2+4+3}{3} ]$$

• var_A:

$$[ \sigma_A^2= \frac{(2-\mu_A)^2+(4-\mu_A)^2+(3-\mu_A)^2}{3} ]$$

• It repeats this independently for students B, C, and D.

The Engine: Z-Score Normalization

Both norm's used Z-Score normalization only.

$$[ \hat{x} = \frac{x-\mu}{\sqrt{\sigma^2+\epsilon}} ]$$

Note: epsilon is added to avoid divide by zero error. Its a very small value.

When to use what?

Layer Norm:

Use LayerNorm when we want to normalize each example independently, without depending on other examples in the batch.

This is especially useful in sequence models, where each token has its own hidden representation.

Generally used in: Transformers, LLMs, RNNs, LSTMs, GRUs, Vision Models.

Batch Norm:

Use BatchNorm when batch-level statistics are meaningful and useful for training.

Generally used in: CNNs, ResNet-style architectures, Image classification, Object detection, Image segmentation.