Executing Notebooks

Course notebooks execute inside Docker containers using papermill for reproducible runs. A central registry (notebook-database.yml) tracks which notebooks exist, their execution environments, and run history.

Execute a single notebook

From the eng-ai-agents repository root:

make execute-notebook NOTEBOOK=transfer-learning/transfer_learning_tutorial.ipynb

This will:

Look up the notebook’s environment in notebook-database.yml
Start the appropriate Docker container via docker compose
Install notebook dependencies (make install-notebooks)
Execute the notebook with papermill, extracting plots and artifacts
Log results to W&B (if WANDB_API_KEY is set in .env)
Update the registry with execution date and duration

Execute all notebooks

Run every registered notebook in sequence:

make execute-all-notebooks

This iterates over all entries in notebook-database.yml, executing each in its registered Docker environment. Notebooks marked as colab are skipped with a printed Colab URL instead.

Make targets

Command	Description
`make execute-notebook NOTEBOOK=<path>`	Execute a single notebook in its registered Docker environment
`make execute-all-notebooks`	Execute all notebooks from the registry
`make install-notebooks`	Install notebook extras and register the Jupyter kernel

Notebook registry

The notebook-database.yml file is the source of truth for all course notebooks. Each entry specifies:

- source: aiml-common/lectures/regression/linear-regression/sgd_sinusoidal_dataset.ipynb
  notebook: regression/linear-regression/sgd_sinusoidal_dataset.ipynb
  code_cells: 10
  environment: torch.dev.gpu
  description: SGD with polynomial regression
  last_executed: 2026-02-08
  duration_seconds: 30.5

Field	Purpose
`source`	Path to the notebook in the eaia documentation repository
`notebook`	Path relative to `notebooks/` in eng-ai-agents
`code_cells`	Number of code cells (for validation)
`environment`	Docker Compose service to use (`torch.dev.gpu`, `torch.dev.cpu`, `colab`)
`description`	Short description of what the notebook covers
`last_executed`	Date of most recent execution
`duration_seconds`	Runtime of last execution

Colab notebooks

Some notebooks require Google Colab (e.g., for Colab-specific APIs or free GPU access). When you run make execute-notebook on a Colab notebook, the pipeline prints the Colab URL instead of executing locally:

═══════════════════════════════════════════════════════════════
  This notebook requires Google Colab
═══════════════════════════════════════════════════════════════

  Open in Colab:
   https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/<path>

Output artifacts

After execution, notebooks produce:

Executed notebook, <stem>-executed.ipynb with all cell outputs populated
PNG images, plots extracted from cell outputs into an output/images/ directory
Plotly HTML, interactive charts saved as HTML files
W&B logs, metrics and images uploaded to the experiment dashboard

Edit this page on GitHub or file an issue.

Semester

Development Environment

Guides

Executing Notebooks

Execute a single notebook

Execute all notebooks

Make targets

Notebook registry

Colab notebooks

Output artifacts

​Execute a single notebook

​Execute all notebooks

​Make targets

​Notebook registry

​Colab notebooks

​Output artifacts

Execute a single notebook

Execute all notebooks

Make targets

Notebook registry

Colab notebooks

Output artifacts