You Only Look Once (YOLO)

YOLOv1 divides the input image into an $S\times S$ grid. If the center of an object falls in cell $i$, that cell is responsible for predicting the object. Each grid cell outputs $B$ bounding-box hypotheses and one set of probabilities over $C$ classes. The resulting tensor has dimensions $S\times S\times (B\cdot 5 + C)$. For VOC, $S=7$, $B=2$, $C=20 \Rightarrow 7\times7\times 30$. Each bounding-box prediction contains five numbers: $$(x, y, w, h, \text{confidence}),$$ Here, $(x,y)$ are the box-center offsets relative to the responsible cell. The values $(w,h)$ are normalized by the image width and height. The confidence target equals the IoU between the predicted box and the closest ground-truth box. It is $0$ when no object is present.

​

Class-specific confidence at test time

During inference, we combine the conditional class probabilities for each cell with the confidence for each box. This gives a score for every box and class: $$\Pr(\text{Class}_i \mid \text{Object}) \cdot \Pr(\text{Object})\cdot \text{IoU}_{\text{pred}}^{\text{truth}} \;=\; \Pr(\text{Class}_i)\cdot \text{IoU}_{\text{pred}}^{\text{truth}}. \tag{1}$$ These class-specific confidence scores are used before NMS.

​

Network architecture and activations

The detector is a single CNN with 24 convolutional layers and 2 fully connected layers. The early layers extract features. The fully connected layers map those features to the $S\times S\times (B\cdot5+C)$ output. A fast variant uses fewer convolutional layers. The final layer uses a linear activation. All other layers use leaky ReLU: $$\phi(x)= \begin{cases} x, & x>0,\\ 0.1x, & \text{otherwise.} \end{cases} \tag{2}$$ Coordinates are normalized as described above.

​

Training targets and responsibility

Each cell predicts $B$ boxes. For a given object, YOLO assigns responsibility to the predictor whose current box has the highest IoU with the ground-truth box. This specialization improves recall. The assignment determines the training targets:

• Only the responsible predictor receives coordinate and objectness regression targets for the object.
• The other predictors in the cell are trained toward “no object” confidence. This reduces spurious positives.

This assignment is made at each iteration using the model’s current boxes.

​

The multi-part loss

YOLOv1 uses a sum-squared error for location, size, objectness (confidence), and classification. The coefficients $\lambda_{\text{coord}}$ and $\lambda_{\text{noobj}}$ balance these terms. To reduce sensitivity to scale, the loss regresses $\sqrt{w},\sqrt{h}$ instead of $w,h$. $$\begin{aligned} \mathcal{L} = \;& \lambda_{\text{coord}} \sum_{i=1}^{S^2} \sum_{j=1}^{B} \mathbf{1}^{\text{obj}}_{ij} \Big[(x_i-\hat{x}_i)^2 + (y_i-\hat{y}_i)^2\Big] \\ &+ \lambda_{\text{coord}} \sum_{i=1}^{S^2} \sum_{j=1}^{B} \mathbf{1}^{\text{obj}}_{ij} \Big[\big(\sqrt{w_i}-\sqrt{\hat{w}_i}\big)^2 + \big(\sqrt{h_i}-\sqrt{\hat{h}_i}\big)^2\Big] \\ &+ \sum_{i=1}^{S^2}\sum_{j=1}^{B} \mathbf{1}^{\text{obj}}_{ij}\big(C_i-\hat{C}_i\big)^2 + \lambda_{\text{noobj}} \sum_{i=1}^{S^2}\sum_{j=1}^{B} \mathbf{1}^{\text{noobj}}_{ij}\big(C_i-\hat{C}_i\big)^2 \\ &+ \sum_{i=1}^{S^2}\mathbf{1}^{\text{obj}}_{i} \sum_{c\in\mathcal{C}}\big(p_i(c)-\hat{p}_i(c)\big)^2. \end{aligned}$$ Here $\mathbf{1}^{\text{obj}}_{ij}=1$ if and only if predictor $j$ in cell $i$ is responsible for an object. The value $\mathbf{1}^{\text{noobj}}_{ij}=1$ for “no object” cases. The coefficients are $\lambda_{\text{coord}}=5$ and $\lambda_{\text{noobj}}=0.5$. The classification loss applies only when a cell contains an object.

​

Optimization details

A typical VOC training schedule uses about 135 epochs, a batch size of 64, momentum of 0.9, and weight decay of $5\!\times\!10^{-4}$. The learning rate warms up from $10^{-3}$ to $10^{-2}$. It then remains at $10^{-2}$ for 75 epochs, $10^{-3}$ for 30 epochs, and $10^{-4}$ for 30 epochs. Regularization includes dropout at a rate of 0.5 after the first fully connected layer. Data augmentation includes random scaling and translation of up to 20%, along with exposure and saturation changes in HSV of up to 1.5×.

​

End-to-end inference

1. Preprocess the image. Resize it, for example to $448\times 448$, and pass it through the CNN once.
2. Decode the raw outputs.
   • For each cell $i$ and predictor $j$, convert the normalized $(x,y,w,h)$ values to image coordinates. Use the predicted confidence $C_{ij}$.
   • Combine the confidence with the class probabilities $p_i(c)$ using Eq. (1). This gives the class-specific scores $s_{ijc} = p_i(c)\cdot C_{ij}$.
3. Filter and suppress the predictions.
   • Discard low-score boxes.
   • Perform non-max suppression for each class. NMS is less important here than in proposal-based pipelines. It removes duplicate predictions from neighboring cells and adds about 2 to 3 mAP points.

​

Strengths and limitations

• YOLO processes the image in one pass and uses information from the entire image. This makes inference very fast.
• Compared with the R-CNN family, YOLO produces fewer background false positives but more localization errors.
• The fixed grid limits performance on crowded scenes with small objects. Downsampling produces coarse features, and small-box localization is sensitive to errors.
• A grid cell is responsible for an object if the object’s center falls inside the cell.
• One predictor learns the geometry of each object assigned to a cell. Responsibility is based on IoU.
• Confidence $=$ objectness $\times$ IoU. Class probabilities apply to the cell. Eq. (1) combines them into a score for each class.
• The loss balances localization, objectness, and classification using $\lambda_{\text{coord}},\lambda_{\text{noobj}}$. It uses the square roots of box dimensions to reduce scale sensitivity. Eq. (3).

​

PyTorch sections

The following sections build a complete YOLOv11 anchor-free detector from scratch in PyTorch.

​

PyTorch reference

Key references: (Redmon et al., 2015; Redmon & Farhadi, 2016; Liu et al., 2015; Canziani et al., 2016; Godard et al., 2016)

​

References

• Canziani, A., Paszke, A., Culurciello, E. (2016). An Analysis of Deep Neural Network Models for Practical Applications. arXiv [cs.CV].
• Godard, C., Aodha, O., Brostow, G. (2016). Unsupervised Monocular Depth Estimation with Left-Right Consistency. arXiv [cs.CV].
• Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., et al. (2015). SSD: Single Shot MultiBox Detector. arXiv [cs.CV].
• Redmon, J., Divvala, S., Girshick, R., Farhadi, A. (2015). You only look once: Unified, real-time object detection. arXiv [cs.CV].
• Redmon, J., Farhadi, A. (2016). YOLO9000: Better, Faster, Stronger. arXiv [cs.CV].

---