> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Regularization in Deep Neural Networks
> Techniques to prevent overfitting in deep neural networks including L2, L1, Dropout, and Early Stopping.
In this chapter we examine training aspects of DNNs and investigate schemes that help avoid overfitting.
## L2 Regularization
The most common form of regularization. It penalizes the squared magnitude of all parameters directly in the objective:
$$
\lambda J_{\text{penalty}} = \lambda \left(\sum_l W_{(l)}^2 \right)
$$
where $l$ is the hidden layer index and $W$ is the weight tensor.
L2 regularization heavily penalizes peaky weight vectors and prefers diffuse weight vectors. Due to multiplicative interactions between weights and inputs, this encourages the network to use all of its inputs a little rather than some of its inputs a lot.
## L1 Regularization
For each weight $w$, add the term $\lambda \mid w \mid$ to the objective. L1 regularization leads weight vectors to become **sparse** during optimization (exactly zero). Neurons with L1 regularization use only a sparse subset of their most important inputs.
**Comparison:**
* **L1**: Sparse weights, feature selection
* **L2**: Diffuse small weights, generally better performance
In practice, if you are not concerned with explicit feature selection, L2 regularization typically gives superior performance.
## Dropout
An extremely effective regularization technique from [Srivastava et al., 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf). During training, dropout keeps a neuron active with probability $p$, or sets it to zero otherwise.
### Standard Dropout
```python theme={null}
p = 0.5 # probability of keeping a unit active
def train_step(X):
H1 = np.maximum(0, np.dot(W1, X) + b1)
U1 = np.random.rand(*H1.shape) < p # dropout mask
H1 *= U1 # drop!
# ... continue forward pass
def predict(X):
H1 = np.maximum(0, np.dot(W1, X) + b1) * p # scale activations
# ... continue forward pass
```
At test time, we scale outputs by $p$ because neurons see all inputs. Expected output during training: $px + (1-p) \cdot 0 = px$.
### Inverted Dropout
Performs scaling at train time, leaving inference untouched:
```python theme={null}
def train_step(X):
H1 = np.maximum(0, np.dot(W1, X) + b1)
U1 = (np.random.rand(*H1.shape) < p) / p # scale during training
H1 *= U1
def predict(X):
H1 = np.maximum(0, np.dot(W1, X) + b1) # no scaling needed
```
Inverted dropout is preferred because prediction code remains unchanged when tuning dropout placement or probability.
Hinton showed that using dropout, dropping out individual neurons during training, leads to a network that is equivalent to averaging over an ensemble of an exponential number of networks.
## Early Stopping
Monitors validation loss during training and stops when validation error increases, retrieving the best model. This acts as an implicit L2 regularizer:
## Practical Recommendations
1. Use a single, global **L2 regularization** strength (cross-validated)
2. Apply **Dropout** after all layers (default $p = 0.5$, tune on validation)
3. Combine with **Batch Normalization** (note: [interesting interference](https://arxiv.org/pdf/1801.05134.pdf) between the two)
4. Use **Early Stopping** with patience parameter
## References
* [Srivastava et al., 2014 - Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
* [Wan et al., 2013 - Regularization of Neural Networks using DropConnect](http://cs.nyu.edu/~wanli/dropc/)
* [Li et al., 2018 - Understanding the Disharmony between Dropout and Batch Normalization](https://arxiv.org/pdf/1801.05134.pdf)
## PyTorch reference
| PyTorch class | Description |
| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [`nn.Dropout`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Dropout.html) | During training, randomly zeroes some of the elements of the input tensor with probability $p$. |
| [`nn.Dropout2d`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Dropout2d.html) | Randomly zero out entire channels. |
***
[Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/optimization/regularization/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).