What Batch Normalization Buys a Detector

​

Grading

​

Competing explanations

Batch normalization can make deep networks easier and faster to train, but its mechanism is less certain. Ioffe and Szegedy, who introduced the method, attributed the improvement to reduced internal covariate shift (Ioffe and Szegedy, 2015). Internal covariate shift occurs when a layer’s input distribution drifts as the weights in earlier layers change during training. Santurkar et al. tested this explanation and proposed another mechanism (Santurkar et al., 2018). They argue that batch normalization smooths the loss landscape. Gradients then remain more predictive from one step to the next, allowing the optimizer to take larger steps. The batch normalization lecture follows the smoothness explanation from Santurkar et al. Its examples use single neurons and one layer. In this assignment, you will test both explanations on Faster R-CNN, a real object detector. You will train it with batch normalization in the feature pyramid network and box head, then train it again without that normalization. Detailed measurements from each run will help you determine which explanation the data support. For more background, read the batch normalization section of Google’s Deep Learning Tuning Playbook. It discusses implementation details, mainly for training across several devices. Those details are not tested in this assignment.

​

What you hand in

• The notebook, with every cell output saved.
• The URL of your Trackio Space, which must be public.

​

What you will need

You need a GPU runtime for this assignment. In Colab, open Runtime, select Change runtime type, and choose a T4 GPU before starting. Each run takes about 30 minutes on a free T4. Runs 1 to 3 are full runs. The extension takes about as long as one additional full run, even if you complete it as two half-length runs. Allow about 2 hours in total. A free Colab session may end before 2 hours. Each run saves its log to a CSV file in the runs folder. Download that folder after each run, and upload it again when you continue in a new session. You also need a Hugging Face account and an access token with write scope. The notebook uses the token for one thing: to create the Hugging Face Space where Trackio shows your training curves.

1. Open your Hugging Face access tokens page and create a token with write scope.
2. In Colab, open the Secrets panel with the key icon in the left sidebar. Add a secret named HF_TOKEN, paste your token as its value, and enable notebook access.

​

Assignment notebook

​

Submitting

Commit the notebook to your private clone of the class repo under assignments/assignment-2. Save every cell output before committing. See the assignment submission guide for more information.

---