> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Executing Notebooks

> Running course notebooks in Docker with automatic output extraction and experiment tracking.

Course notebooks execute inside [Docker containers](/aiml-common/resources/environment/docker-containers) using `papermill` for reproducible runs. A central registry (`notebook-database.yml`) tracks which notebooks exist, their execution environments, and run history.

## Execute a single notebook

From the [eng-ai-agents](https://github.com/pantelis/eng-ai-agents) repository root:

```bash theme={null}
make execute-notebook NOTEBOOK=transfer-learning/transfer_learning_tutorial.ipynb
```

This will:

1. Look up the notebook's environment in `notebook-database.yml`
2. Start the appropriate Docker container via `docker compose`
3. Install notebook dependencies (`make install-notebooks`)
4. Execute the notebook with `papermill`, extracting plots and artifacts
5. Log results to [W\&B](/aiml-common/resources/environment/experiment-manager) (if `WANDB_API_KEY` is set in `.env`)
6. Update the registry with execution date and duration

## Execute all notebooks

Run every registered notebook in sequence:

```bash theme={null}
make execute-all-notebooks
```

This iterates over all entries in `notebook-database.yml`, executing each in its registered Docker environment. Notebooks marked as `colab` are skipped with a printed Colab URL instead.

## Make targets

| Command                                 | Description                                                    |
| --------------------------------------- | -------------------------------------------------------------- |
| `make execute-notebook NOTEBOOK=<path>` | Execute a single notebook in its registered Docker environment |
| `make execute-all-notebooks`            | Execute all notebooks from the registry                        |
| `make install-notebooks`                | Install notebook extras and register the Jupyter kernel        |

## Notebook registry

The `notebook-database.yml` file is the source of truth for all course notebooks. Each entry specifies:

```yaml theme={null}
- source: aiml-common/lectures/regression/linear-regression/sgd_sinusoidal_dataset.ipynb
  notebook: regression/linear-regression/sgd_sinusoidal_dataset.ipynb
  code_cells: 10
  environment: torch.dev.gpu
  description: SGD with polynomial regression
  last_executed: 2026-02-08
  duration_seconds: 30.5
```

| Field              | Purpose                                                                   |
| ------------------ | ------------------------------------------------------------------------- |
| `source`           | Path to the notebook in the eaia documentation repository                 |
| `notebook`         | Path relative to `notebooks/` in eng-ai-agents                            |
| `code_cells`       | Number of code cells (for validation)                                     |
| `environment`      | Docker Compose service to use (`torch.dev.gpu`, `torch.dev.cpu`, `colab`) |
| `description`      | Short description of what the notebook covers                             |
| `last_executed`    | Date of most recent execution                                             |
| `duration_seconds` | Runtime of last execution                                                 |

## Colab notebooks

Some notebooks require Google Colab (e.g., for Colab-specific APIs or free GPU access). When you run `make execute-notebook` on a Colab notebook, the pipeline prints the Colab URL instead of executing locally:

```
═══════════════════════════════════════════════════════════════
  This notebook requires Google Colab
═══════════════════════════════════════════════════════════════

  Open in Colab:
   https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/<path>
```

## Output artifacts

After execution, notebooks produce:

* **Executed notebook**, `<stem>-executed.ipynb` with all cell outputs populated
* **PNG images**, plots extracted from cell outputs into an `output/images/` directory
* **Plotly HTML**, interactive charts saved as HTML files
* **W\&B logs**, metrics and images uploaded to the [experiment dashboard](https://wandb.ai/pantelis/eng-ai-agents/workspace)

***

<Callout icon="pen-to-square" iconType="regular">
  [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/resources/environment/executing-notebooks.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).
</Callout>