Gradient Descent & Optimization Algorithms

Gradient descent is the foundational algorithm for training machine learning models. The goal is to minimize a loss function $L(\theta)$ — a measure of how wrong the model's predictions are — by iteratively adjusting the model parameters $\theta$ in the direction that reduces the loss.

The Core Idea

The gradient $\nabla_\theta L$ points in the direction of steepest increase of the loss. To minimize, we move in the opposite direction:

$\theta \leftarrow \theta - \eta \nabla_\theta L(\theta)$

where $\eta$ (eta) is the learning rate — how big a step we take each iteration.

# Bare-bones gradient descent
for epoch in range(num_epochs):
    loss = compute_loss(model, data)
    loss.backward()          # compute gradients
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad
            param.grad.zero_()

Variants

Batch Gradient Descent

Uses the entire dataset to compute the gradient at each step.

Accurate gradient estimate
Very slow and memory-intensive for large datasets
Not practical for most modern ML workloads

Stochastic Gradient Descent (SGD)

Uses a single random sample per update.

$\theta \leftarrow \theta - \eta \nabla_\theta L(\theta; x^{(i)}, y^{(i)})$

Fast updates, can escape local minima due to noise
High variance — loss oscillates significantly during training
Can be slow to converge near the optimum

Mini-Batch SGD

Uses a small batch (typically 32-512 samples). This is the standard in practice.

Balances speed and gradient accuracy
Leverages GPU parallelism efficiently
The "SGD" in most deep learning frameworks refers to mini-batch SGD

Challenges

Challenge	Description
Learning rate too high	Loss diverges or oscillates
Learning rate too low	Convergence is extremely slow
Saddle points	Gradient is zero but not at a minimum
Vanishing gradients	Gradients near zero in early layers — common with sigmoid/tanh
Exploding gradients	Gradients grow uncontrollably — common in RNNs

Momentum

Momentum accumulates a velocity vector in directions of consistent gradient, dampening oscillations:

$v \leftarrow \beta v - \eta \nabla_\theta L$ $\theta \leftarrow \theta + v$

A typical value is $\beta = 0.9$ . Nesterov momentum computes the gradient at the "looked-ahead" position, which often converges faster.

Adaptive Optimizers

AdaGrad

Scales the learning rate per parameter by the sum of squared past gradients. Parameters with large gradients get smaller updates.

Good for sparse data (NLP, recommendation)
Learning rate shrinks monotonically — can become too small

RMSProp

Uses an exponentially decaying average of squared gradients to avoid AdaGrad's monotonic decay:

$G \leftarrow \rho G + (1-\rho)(\nabla L)^2$ $\theta \leftarrow \theta - \frac{\eta}{\sqrt{G + \epsilon}} \nabla L$

Adam (Adaptive Moment Estimation)

Combines momentum (first moment) and RMSProp (second moment) with bias correction. The default choice for most deep learning tasks.

$m \leftarrow \beta_1 m + (1 - \beta_1) g$ $v \leftarrow \beta_2 v + (1 - \beta_2) g^2$ $\hat{m} = \frac{m}{1 - \beta_1^t}, \quad \hat{v} = \frac{v}{1 - \beta_2^t}$ $\theta \leftarrow \theta - \frac{\eta}{\sqrt{\hat{v}} + \epsilon} \hat{m}$

Default hyperparameters: $\beta_1 = 0.9$ , $\beta_2 = 0.999$ , $\epsilon = 10^{-8}$ , $\eta = 10^{-3}$ .

import torch.optim as optim
 
optimizer = optim.Adam(model.parameters(), lr=1e-3)
 
for batch in dataloader:
    optimizer.zero_grad()
    loss = criterion(model(batch['x']), batch['y'])
    loss.backward()
    optimizer.step()

Learning Rate Scheduling

A fixed learning rate is rarely optimal. Common schedules:

Schedule	Behavior
Step decay	Reduce by factor every N epochs
Cosine annealing	Smoothly decays following a cosine curve
Warmup + decay	Start with small LR, ramp up, then decay — used in Transformers
Reduce on plateau	Reduce when validation loss stops improving

scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
 
for epoch in range(100):
    train(...)
    scheduler.step()

Comparison Summary

Optimizer	Adaptive LR	Momentum	When to Use
SGD	No	Optional	CNNs, when fine-tuned carefully
SGD + Momentum	No	Yes	Computer vision, ResNets
Adam	Yes	Yes	Default for NLP, Transformers
AdamW	Yes	Yes	Recommended for fine-tuning LLMs
RMSProp	Yes	Optional	RNNs, non-stationary problems

AdamW is Adam with decoupled weight decay — it is the standard optimizer for training Transformers and fine-tuning language models.

Gradient Descent & Optimization Algorithms in Machine Learning