Regularization in Machine Learning

Regularization refers to techniques that prevent a model from overfitting — memorizing training data instead of learning to generalize to unseen examples.

Overfitting occurs when a model is too complex relative to the amount of training data: it fits the noise in the training set and performs poorly on new data.

The Bias-Variance Tradeoff

At the heart of regularization is the bias-variance tradeoff:

• High bias (underfitting): model is too simple, fails to capture the signal.
• High variance (overfitting): model is too complex, captures noise.

Regularization reduces variance at the cost of a slight increase in bias.

Training loss:    low  ←————————→  high
Validation loss:  high ←————————→  optimal ←————→ high
                  (overfit)        (just right)   (underfit)

L2 Regularization (Ridge / Weight Decay)

Adds the squared magnitude of all weights to the loss function:

L_{\text{reg}} = L + \frac{\lambda}{2} \sum_i w_i^2

• Encourages small weights, penalizing large values uniformly.
• Weights shrink toward zero but never exactly reach zero.
• Called weight decay in the context of neural networks.

import torch.optim as optim
 
# Weight decay is L2 regularization in PyTorch
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)

In scikit-learn:

from sklearn.linear_model import Ridge
 
model = Ridge(alpha=1.0)  # alpha = lambda, the regularization strength
model.fit(X_train, y_train)

L1 Regularization (Lasso)

Adds the absolute value of weights to the loss:

L_{\text{reg}} = L + \lambda \sum_i |w_i|

• Produces sparse models — many weights become exactly zero.
• Useful for feature selection: irrelevant features get eliminated.
• L1 is not differentiable at zero, but subgradients handle this in practice.

from sklearn.linear_model import Lasso
 
model = Lasso(alpha=0.1)
model.fit(X_train, y_train)

L1 vs L2

Elastic Net combines both: \lambda_1 |w| + \lambda_2 w^2.

Dropout

Dropout (Srivastava et al., 2014) randomly sets a fraction p of neurons to zero during each training forward pass. This prevents neurons from co-adapting and forces the network to learn redundant representations.

• At inference, all neurons are active and outputs are scaled by (1-p).
• Effectively trains an ensemble of 2^n different sub-networks.
• Common rates: p = 0.1 to 0.5 (use lower values for larger models).

import torch.nn as nn
 
model = nn.Sequential(
    nn.Linear(512, 256),
    nn.ReLU(),
    nn.Dropout(p=0.3),   # 30% of neurons dropped during training
    nn.Linear(256, 10),
)
 
model.train()   # dropout is active
model.eval()    # dropout is disabled

Batch Normalization

Batch Normalization (Ioffe & Szegedy, 2015) normalizes activations within each mini-batch, reducing internal covariate shift.

\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}

Then applies learnable scale and shift: y = \gamma \hat{x} + \beta.

• Acts as a regularizer — often reduces or eliminates the need for Dropout.
• Allows higher learning rates and speeds up training significantly.
• Standard in CNNs; less common in Transformers (which use Layer Normalization).

model = nn.Sequential(
    nn.Linear(512, 256),
    nn.BatchNorm1d(256),
    nn.ReLU(),
    nn.Linear(256, 10),
)

Early Stopping

Monitor validation loss during training and stop when it starts increasing — even if training loss continues to decrease.

best_val_loss = float('inf')
patience = 10
no_improve = 0
 
for epoch in range(max_epochs):
    train_loss = train_one_epoch(model, train_loader)
    val_loss = evaluate(model, val_loader)
 
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        torch.save(model.state_dict(), 'best_model.pt')
        no_improve = 0
    else:
        no_improve += 1
        if no_improve >= patience:
            print(f'Early stopping at epoch {epoch}')
            break
 
model.load_state_dict(torch.load('best_model.pt'))

Data Augmentation

For image and text tasks, artificially expanding the training set by applying transformations is one of the most effective forms of regularization:

from torchvision import transforms
 
train_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomCrop(32, padding=4),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(),
    transforms.Normalize(mean, std),
])

Summary

Further Reading

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava et al., 2014)
• Batch Normalization (Ioffe & Szegedy, 2015)
• An Introduction to Statistical Learning (James et al.) — Chapter 6 covers shrinkage methods
• CS231n — Neural Networks Part 2: Setting up the Data and the Loss