> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# UAV Drone Detection and Tracking
> Detect and track UAV drones in video using a deep learning object detector and a Kalman filter.
* Drone object detection, 50
* Kalman filter tracking, 50
## Overview and learning objectives
Multi-Object Tracking (MOT) is a core visual ability that humans use to perform kinetic tasks and coordinate activities in dynamic environments. The AI community has recognized the importance of MOT through a series of [competitions](https://motchallenge.net).
In this assignment, the target object class is `drone`. You will detect drones in video footage and track them using [Kalman filters](https://en.wikipedia.org/wiki/Kalman_filter). The assignment situates probabilistic reasoning in the physical security domain.
By completing this assignment, you will:
* Identify and use a drone-specific object detection dataset.
* Fine-tune or configure a deep learning detector for the `drone` class.
* Implement a Kalman filter to track detections across frames.
* Visualize 2D trajectories superimposed on video.
***
## Test videos
The following two videos are your primary test inputs. Download them locally before starting.
Use [yt-dlp](https://github.com/yt-dlp/yt-dlp) to download them:
```bash macOS theme={null}
brew install ffmpeg yt-dlp
```
```bash Ubuntu / WSL theme={null}
sudo apt update && sudo apt install -y ffmpeg
pip install yt-dlp
```
```bash theme={null}
yt-dlp -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]" \
-o "drone_video_1.mp4" \
"https://www.youtube.com/watch?v=DhmZ6W1UAv4"
```
Repeat for the second video.
```bash theme={null}
mkdir -p frames
ffmpeg -i drone_video_1.mp4 -vf "fps=5" frames/frame_%04d.jpg
```
Sampling at 5 fps is a reasonable starting point. Adjust based on drone speed.
***
## Task 1: Drone object detection (50 points)
### Dataset
Find a dataset that contains labeled `drone` bounding boxes. Be careful to distinguish between:
* Datasets that detect objects **from** drones (aerial imagery), not what you want.
* Datasets that detect **the drone itself**, what you want.
Resources to search:
* [Roboflow Universe, Drone Detection](https://universe.roboflow.com/search?q=drone+detection)
* [VisDrone](https://github.com/VisDrone/VisDrone-Dataset) (includes drone class)
* [Hugging Face Datasets Hub](https://huggingface.co/datasets)
Prefer datasets stored in Parquet or standard COCO/YOLO formats for ease of loading.
### Detector
You must use a deep learning model. You may use a pretrained architecture, fine-tuning is encouraged but not required. Recommended starting points:
* [Ultralytics YOLOv8](https://docs.ultralytics.com/), straightforward fine-tuning API
* [RT-DETR](https://docs.ultralytics.com/models/rtdetr/), transformer-based, strong accuracy
### Deliverable
Split each video into frames and run your detector on every frame. Save all frames that contain at least one detection to a folder called `detections/`. Write your code so that it processes all `.mp4` files in a given directory, not just the two test videos.
***
## Task 2: Kalman filter tracking (50 points)
Use the [`filterpy`](https://filterpy.readthedocs.io/en/latest/kalman/KalmanFilter.html) library to implement a Kalman filter that tracks the drone across frames. Initialize the filter with detections from Task 1.
Your state vector should represent at minimum the 2D pixel position of the drone bounding box center (and optionally its velocity). For each track:
1. **Predict** the next state using the Kalman filter motion model.
2. **Update** the state using the detector output for that frame.
3. **Handle missing detections**, the filter must continue predicting even when the detector misses the drone for a small number of consecutive frames.
### Deliverable
Produce one output video per input video. Each output video must contain **only the frames where the drone is present** and must overlay:
* The detector bounding box.
* The 2D trajectory as a polyline connecting the tracker-estimated center positions across frames.
Use `ffmpeg` and OpenCV to compose the output.
***
## Evaluation criteria
| Criterion | Description |
| ------------------------ | ---------------------------------------------------------------------------------- |
| Detection quality | Are detections consistent and semantically correct (drone class, not background)? |
| Tracker correctness | Does the Kalman filter correctly predict and update across frames? |
| Trajectory visualization | Is the 2D trajectory clearly superimposed on the output video? |
| Code generality | Does the pipeline process any directory of `.mp4` files, not just the test videos? |
| Report clarity | Can you explain your detector choice, filter design, and failure cases? |
***
## Deliverables
1. A **Hugging Face dataset** containing the `detections/` sample frames (Parquet format).
2. **Output tracking videos** (one per test input) uploaded to your personal YouTube channel and embedded in your README.
3. A **`README.md`** in your submission repository covering:
* Dataset choice and detector configuration.
* Kalman filter state design and noise parameters.
* Failure cases and how the tracker handles missed detections.
***
[Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/assignments/main/ai-spring-2026/assignment-3.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).