> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scalable Inference with Ray and Triton

> Leveraging Ray and Triton for Concurrent End-to-End Inference Job Processing.

## Introduction

This document outlines a distributed, actor-based inference system designed to process large-scale geospatial machine learning workloads from raw imagery to vectorized outputs.

The system combines **Ray** for parallel and stateful pipeline orchestration with **NVIDIA's Triton Inference Server** for efficient GPU-accelerated model serving, together enabling **high throughput** with minimal idle time across all stages.

### About Ray

[Ray](https://www.ray.io/) is an open-source framework for building and running distributed applications at scale. It provides a unified runtime for **tasks** (stateless units of work) and **actors** (stateful, long-lived processes). Ray was chosen because it:

* Supports **persistent actors** that maintain state across calls
* Offers a simple API for **asynchronous and parallel execution**
* Can scale from a **single machine to a multi-node cluster**

### About Triton Inference Server

[NVIDIA Triton Inference Server](https://developer.nvidia.com/nvidia-triton-inference-server) enables deployment of trained AI models from multiple frameworks. Triton was chosen because it:

* Provides **optimized GPU utilization** through dynamic batching
* Supports **multiple model backends**
* Integrates with both local and cloud deployments

## High-Level Ray Actor Overview

| Actor(s)                 | Primary Function                     | Key Inputs         | Key Outputs                  |
| ------------------------ | ------------------------------------ | ------------------ | ---------------------------- |
| ControllerActor          | Orchestrates all stages              | Job list, config   | Run metadata, progress table |
| TileLoaderActor          | Ingests tiles, extracts chips        | GeoTIFF tiles      | Chips + metadata             |
| InputQueueActor          | Buffers chips between stages         | Chip records       | Chip records for batching    |
| InferenceDispatcherActor | Batches chips, runs inference        | Chips from queue   | Chip predictions             |
| AggregatorActor          | Buffers predictions per tile         | Chip predictions   | Complete per-tile sets       |
| PostProcessingActor      | Stitches masks, extracts centerlines | Complete tile sets | GeoJSON centerlines          |
| CenterlineWorker         | Converts polygons to centerlines     | Polygon batches    | Vectorized centerlines       |

## Pipeline Ingress

### Building Tile Jobs

The `build_tile_jobs(...)` function creates a standardized list of **per-tile job dicts**:

```python theme={null}
{
    "tif_path": "s3://bucket/path/to.tif",
    "tile_id": "H6B10",
    "job_id": "njogis-2020",
    "requested_chip_size": 256,
    "requested_chip_overlap": 32,
    "use_pseudo_color_nir": True,
    "target_format": "NIR-GB",
    "target_model_input_size": [3, 256, 256]
}
```

### Running the Pipeline

```python theme={null}
store = s3_store()
files = list_s3_files(store, prefix="imagery/njogis-tiles/2020/cog")
tif_keys = sorted(files["key"])
s3_tif_paths = [os.path.join("s3://njtpa/", k) for k in tif_keys]

tile_jobs = build_tile_jobs(tif_paths=s3_tif_paths, job_id="njogis-tiles_2020")

main(
    tile_jobs=tile_jobs,
    run_id="njogis-tiles_2020_cog_full_run",
    endpoint="triton-inference-server:8001",
    model_name="batched_semseg_model",
    model_version="1",
    num_tileloaders=3,
    num_postprocessors=3,
    storage_mode="s3",
    store=store,
)
```

## Pipeline Stages

### Lifecycle of a Single Tile

1. **Ingestion** – `TileLoaderActor` reads the tile, extracts chips, sends to queue
2. **Queuing** – `InputQueueActor` buffers chips for downstream consumption
3. **Inference** – `InferenceDispatcherActor` batches chips, runs model inference
4. **Aggregation** – `AggregatorActor` groups predictions until tile complete
5. **Post-Processing** – `PostProcessingActor` stitches mask, extracts centerlines

### Stage 0, ControllerActor

* **Startup & Wiring:** Launches all workers, connects handoffs
* **Progress Tracking:** Maintains progress table indexed by (job\_id, tile\_id)
* **Health & Logging:** Polls actors, logs status summaries
* **Completion Criteria:** Declares complete when all tiles processed

### Stage 1, TileLoaderActor

* Reads `.tif` tiles from local or S3 storage
* Extracts geospatial metadata (CRS, transform, dimensions)
* Splits tiles into chips with overlap
* Assigns composite keys for traceability

### Stage 2, InferenceDispatcherActor

* Accumulates chips into batches (default size: 200)
* Normalizes inputs (mean/std from model config)
* Sends mini-batches to Triton via gRPC
* Applies softmax and confidence thresholding
* Handles backpressure with exponential backoff

### Stage 3, PostProcessingActor

* Reconstructs full-size prediction mask from chips
* Applies morphological operations
* Converts polygons to centerlines via CenterlineWorker pool
* Writes GeoJSON output to local/S3

## Triton Inference Server Configuration

### Model Directory Structure

```
models/
└── batched_semseg_model/
    ├── config.pbtxt
    └── 1/
        └── model.onnx
```

### Configuration

```yaml theme={null}
name: "batched_semseg_model"
platform: "onnxruntime_onnx"
max_batch_size: 200

instance_group [
  {
    kind: KIND_GPU
    count: 1
    gpus: [0]
  }
]

dynamic_batching {
  preferred_batch_size: [16, 32, 64, 128, 150, 200]
  max_queue_delay_microseconds: 100000
  preserve_ordering: true
}

input [
  {
    name: "image"
    data_type: TYPE_FP32
    dims: [3, -1, -1]
  }
]

output [
  {
    name: "sem_seg"
    data_type: TYPE_FP32
    dims: [2, -1, -1]
  }
]
```

## Chapter Summary

The inference pipeline transforms large-scale geospatial imagery into usable vector data through a fully automated, parallel workflow:

1. Starting from GeoTIFF tiles
2. Applying configurable preprocessing
3. Performing semantic segmentation via Triton
4. Reassembling predictions at tile scale
5. Converting to vectorized sidewalk centerlines

The Ray-based architecture provides:

* **Scalable concurrency** across multiple tiles and jobs
* **Robust fault handling** with per-tile tracking
* **Flexible deployment** for local or cloud environments
* **Minimal idle time** through asynchronous handoffs
* **Clear observability** via central controller

Each run produces:

* Vectorized per-tile GeoJSON centerlines
* Progress and status tracking tables
* Complete run metadata and execution logs

***

<Callout icon="pen-to-square" iconType="regular">
  [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/products/applications/remote-sensing/inference/pipeline_structure/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).
</Callout>