> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Flow matching on a 1D mixture

> Learning the transport field from samples, with three couplings.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/flow-matching/mog-1d/index.ipynb)

```python theme={null}
import numpy as np
import matplotlib.pyplot as plt
from scipy import sparse, stats
from scipy.optimize import linprog
from scipy.special import logsumexp
from scipy.stats import norm
```

```python theme={null}
# Source p_0 is the standard normal N(0, 1). Target p_1 is this Gaussian mixture.
PI = np.array([0.3, 0.5, 0.2])
MU = np.array([-2.0, 1.0, 3.5])
SIGMA = np.array([0.4, 0.6, 0.3])
U_CLIP = 1e-12  # quantile levels are clipped to [U_CLIP, 1 - U_CLIP]


def mixture_pdf(x, pi=PI, mu=MU, sigma=SIGMA):
    x = np.asarray(x, dtype=float)[..., None]
    return np.sum(pi * norm.pdf(x, mu, sigma), axis=-1)


def mixture_cdf(x, pi=PI, mu=MU, sigma=SIGMA):
    x = np.asarray(x, dtype=float)[..., None]
    return np.sum(pi * norm.cdf(x, mu, sigma), axis=-1)


def mixture_ppf(u, pi=PI, mu=MU, sigma=SIGMA, iters=64):
    """Quantile function F_1^{-1}(u) by vectorized bisection on the mixture CDF.

    The root lies between the smallest and the largest component quantile at level u.
    """
    u = np.clip(np.asarray(u, dtype=float), U_CLIP, 1 - U_CLIP)
    q = mu + sigma * norm.ppf(u)[..., None]
    lo, hi = q.min(axis=-1), q.max(axis=-1)
    for _ in range(iters):
        mid = 0.5 * (lo + hi)
        below = mixture_cdf(mid, pi, mu, sigma) < u
        lo = np.where(below, mid, lo)
        hi = np.where(below, hi, mid)
    return 0.5 * (lo + hi)


def monge_map(x0, pi=PI, mu=MU, sigma=SIGMA):
    """Increasing map T = F_1^{-1} o F_0 from N(0, 1) to the mixture."""
    return mixture_ppf(norm.cdf(x0), pi, mu, sigma)


def w2_squared(n_quad=20000, pi=PI, mu=MU, sigma=SIGMA):
    """W_2^2(p_0, p_1) as the integral of (F_0^{-1} - F_1^{-1})^2 over (0, 1), midpoint rule."""
    u = (np.arange(n_quad) + 0.5) / n_quad
    return np.mean((norm.ppf(u) - mixture_ppf(u, pi, mu, sigma)) ** 2)


def fm_path_density(t, x, pi=PI, mu=MU, sigma=SIGMA):
    """Density of x_t = (1 - t) x_0 + t x_1 with x_0 ~ N(0, 1) and x_1 ~ p_1 independent."""
    s = np.sqrt((1 - t) ** 2 + t**2 * sigma**2)
    return mixture_pdf(x, pi, t * mu, s)


def plot_transport(ax, t_grid, x_grid, density, trajectories=None, title="",
                   action=None, edge=0.22):
    """Standard figure: heatmap of p_t(x) with time on the horizontal axis.

    density has shape (len(x_grid), len(t_grid)). The source and target columns are
    drawn as profiles outside the left and right edges. trajectories, if given, has
    shape (len(t_grid), n_particles). action, if given, is reported in the title.
    """
    vmax = np.quantile(density, 0.995)
    ax.pcolormesh(t_grid, x_grid, density, shading="auto", cmap="magma", vmin=0, vmax=vmax)
    for col, side, color in ((0, -1, "tab:blue"), (-1, 1, "tab:orange")):
        profile = density[:, col] / density[:, col].max()
        t_edge = t_grid[col]
        ax.fill_betweenx(x_grid, t_edge, t_edge + side * edge * profile,
                         color=color, alpha=0.7, lw=0)
    if trajectories is not None:
        ax.plot(t_grid, trajectories, color="white", lw=0.6, alpha=0.7)
    ax.set_xlim(t_grid[0] - 1.05 * edge, t_grid[-1] + 1.05 * edge)
    ax.set_ylim(x_grid[0], x_grid[-1])
    ax.set_xticks(np.linspace(t_grid[0], t_grid[-1], 5))
    ax.set_xlabel("t")
    ax.set_ylabel("x")
    if action is not None:
        title = f"{title} (kinetic action {action:.3f})" if title else f"kinetic action {action:.3f}"
    ax.set_title(title)
```

## Learning the field from samples

The [dynamic transport section](https://aegean.ai/aiml-common/lectures/optimal-transport/dynamic-transport/index.md) moved the standard normal $p_0$ to the mixture $p_1$ along the geodesic, and it could compute everything exactly: the density $p_t$, its CDF $F_t$ and the field $v_t$. Two facts from that section carry over. A field $v_t$ moves a path $p_t$ when the pair satisfies the continuity equation $\partial_t p_t + \partial_x (p_t v_t) = 0$. In 1D the path fixes the field, $v_t(x) = -\partial_t F_t(x) / p_t(x)$.

All of this was possible because the mixture was built by hand, so its density and CDF are known. For real data such as images you have samples of $p_1$ and nothing else: no density, no CDF, no field. Flow matching learns a field from samples alone (Lipman et al., 2022). The general definitions are on the [flow matching page](https://aegean.ai/aiml-common/lectures/flow-matching/index.md). This section runs flow matching on the same mixture, where every learned quantity can still be compared with an exact reference.

The code below repeats the grids and the geodesic functions of the dynamic transport section.

```python theme={null}
from scipy.integrate import quad
from scipy.interpolate import CubicSpline

T_GRID = np.linspace(0, 1, 201)
X_GRID = np.linspace(-6, 7, 2001)

# Fine grid of source points for pushing densities forward. Beyond |x_0| = 6.5 the
# quantile level Phi(x_0) approaches U_CLIP and T(x_0) stops being accurate.
X0_FINE = np.linspace(-6.5, 6.5, 20001)
T_FINE = monge_map(X0_FINE)
T_PRIME_FINE = norm.pdf(X0_FINE) / mixture_pdf(T_FINE)   # T'(x_0) = p_0(x_0) / p_1(T(x_0))
LEVELS = np.linspace(0.02, 0.98, 25)                     # quantile levels of plotted particles

def _source_point(t, x):
    """The source point x_0 whose straight path reaches x at time t (x is clipped to the range)."""
    x_t = (1 - t) * X0_FINE + t * T_FINE
    return CubicSpline(x_t, X0_FINE)(np.clip(x, x_t[0], x_t[-1]))


def geodesic_density(t, x):
    """Density of x_t = (1 - t) x_0 + t T(x_0), pushed forward from the fine source grid."""
    x_t = (1 - t) * X0_FINE + t * T_FINE
    p = norm.pdf(X0_FINE) / (1 - t + t * T_PRIME_FINE)
    # Cubic, not linear, interpolation: where the map stretches the grid, linear
    # interpolation errs by about 0.1% and the 1D field formula divides by p_t.
    inside = (x >= x_t[0]) & (x <= x_t[-1])
    return np.where(inside, CubicSpline(x_t, p)(np.clip(x, x_t[0], x_t[-1])), 0.0)


def geodesic_cdf(t, x):
    """F_t(x) = F_0(x_0) for the source point x_0 that reaches x at time t."""
    return norm.cdf(_source_point(t, x))


def geodesic_field(t_grid, x_grid):
    """v_t(x) = T(x_0) - x_0 at the source point of x, shape (len(x_grid), len(t_grid))."""
    v = np.empty((len(x_grid), len(t_grid)))
    for j, t in enumerate(t_grid):
        x0 = _source_point(t, x_grid)
        v[:, j] = monge_map(x0) - x0
    return v


def kinetic_action(density_fn, field, t_grid, x_grid):
    """Integral of p_t v_t^2 over x and t by the trapezoid rule, skipping nan cells of field."""
    p = np.stack([density_fn(t, x_grid) for t in t_grid], axis=1)
    integrand = np.where(np.isfinite(field), p * np.nan_to_num(field) ** 2, 0.0)
    return np.trapezoid(np.trapezoid(integrand, x_grid, axis=0), t_grid)

def field_from_cdf(cdf_fn, density_fn, t_grid, x_grid, dt=1e-4):
    """v_t(x) = -dF_t(x)/dt / p_t(x) on the grid, nan where p_t(x) <= 1e-3.

    Central differences in t inside (0, 1); second-order one-sided differences at t = 0
    and t = 1, so the path is never evaluated outside [0, 1].
    """
    v = np.empty((len(x_grid), len(t_grid)))
    for j, t in enumerate(t_grid):
        F = lambda s: cdf_fn(s, x_grid)
        if t - dt < 0:
            dF = (-3 * F(t) + 4 * F(t + dt) - F(t + 2 * dt)) / (2 * dt)
        elif t + dt > 1:
            dF = (3 * F(t) - 4 * F(t - dt) + F(t - 2 * dt)) / (2 * dt)
        else:
            dF = (F(t + dt) - F(t - dt)) / (2 * dt)
        p = density_fn(t, x_grid)
        v[:, j] = np.where(p > 1e-3, -dF / np.maximum(p, 1e-3), np.nan)
    return v
```

## A path you can sample

Flow matching builds its path from pairs. Draw $x_0 \sim p_0$ and $x_1 \sim p_1$ independently and move along the segment between them,

$$
x_t = (1 - t)\,x_0 + t\,x_1 .
$$

Samples of this path cost nothing: draw a pair and a time. The density is also known for the mixture. Given that $x_1$ comes from component $k$, $x_t$ is a sum of two independent Gaussians, so it is Gaussian with mean $t\mu_k$ and variance

$$
s_k^2(t) = (1 - t)^2 + t^2 \sigma_k^2 .
$$

So $p_t$ is again a mixture, with the same weights $\pi_k$, means $t\mu_k$ and widths $s_k(t)$. Its CDF is the same mixture of normal CDFs.

Compare the width with the geodesic between $\mathcal{N}(0, 1)$ and a single component $\mathcal{N}(\mu_k, \sigma_k^2)$. There the width interpolates in a straight line, $(1 - t) + t\sigma_k$. The independent path is narrower at every time in between. Each component shrinks below its final width $\sigma_k$ and then widens to $\sigma_k$ at $t = 1$.

```python theme={null}
def _fm_scales(t, sigma=SIGMA):
    """Widths s_k(t) and their time derivatives for the independent-pair path."""
    s = np.sqrt((1 - t) ** 2 + t**2 * sigma**2)
    s_dot = (t * sigma**2 - (1 - t)) / s
    return s, s_dot


def fm_path_cdf(t, x):
    """CDF of x_t = (1 - t) x_0 + t x_1 for independent x_0 ~ N(0, 1) and x_1 ~ p_1."""
    s, _ = _fm_scales(t)
    return mixture_cdf(x, PI, t * MU, s)
```

```python theme={null}
t_inner = T_GRID[1:-1, None]
s_inner, _ = _fm_scales(t_inner)
assert np.all(s_inner < (1 - t_inner) + t_inner * SIGMA)       # narrower than the straight widths
assert np.all(s_inner.min(axis=0) < SIGMA)                     # each component dips below sigma_k
```

![Line plot of component width against t for the three mixture components. Solid curves for the independent-pair path start at 1, fall below their final widths 0.4, 0.6 and 0.3, and rise back to them at t = 1. Dashed straight lines from 1 to each final width lie above the solid curves at every interior time.](https://mintcdn.com/aegeanaiinc/TxnGZcYFzPhEZiXx/aiml-common/lectures/flow-matching/mog-1d/images/cell_6_output_1.png?fit=max\&auto=format\&n=TxnGZcYFzPhEZiXx\&q=85\&s=850f323bbbc9fa1c3a43c298ba9879b5)

*Width of each mixture component along the independent-pair path (solid) and along the straight interpolation of widths (dashed). Each solid curve falls below its final value before it returns to it at $t = 1$.*

The standard figure puts the independent-pair path next to the geodesic. The field of each path comes from the 1D formula of the dynamic transport section, and the title of each panel reports its kinetic action.

```python theme={null}
x_plot = np.linspace(-4.5, 5.5, 401)
v_fm_cdf = field_from_cdf(fm_path_cdf, fm_path_density, T_GRID, X_GRID)
action_fm_cdf = kinetic_action(fm_path_density, v_fm_cdf, T_GRID, X_GRID)
v_geo = geodesic_field(T_GRID, X_GRID)
action_geo = kinetic_action(geodesic_density, v_geo, T_GRID, X_GRID)
w2 = w2_squared()
print(f"kinetic action, independent pairs: {action_fm_cdf:.4f} ; geodesic: {action_geo:.4f} ; "
      f"W_2^2: {w2:.4f}")
```

```output theme={null}
kinetic action, independent pairs: 2.4474 ; geodesic: 1.5149 ; W_2^2: 1.5149
```

![Two standard transport heatmaps side by side. Left, the independent-pair path, with kinetic action 2.447: the source bump narrows until about t = 0.3 and splits into three bands after t = 0.5. Right, the geodesic, with kinetic action 1.515: the bump splits into three straight bands right at t = 0.](https://mintcdn.com/aegeanaiinc/TxnGZcYFzPhEZiXx/aiml-common/lectures/flow-matching/mog-1d/images/cell_8_output_1.png?fit=max\&auto=format\&n=TxnGZcYFzPhEZiXx\&q=85\&s=020975cf33e5bfa08d5badd1c8fb7be3)

*Left, the independent-pair path of flow matching. Right, the geodesic of the dynamic transport section. Both join the same endpoints.*

The independent-pair path keeps $p_0$ in one piece for longer. It narrows first and splits into the three components late. The geodesic splits right away. The independent path also costs more, and the last subsections measure by how much.

## Conditional paths and the marginal field

Each pair moves along its own straight line with constant velocity $x_1 - x_0$. These conditional paths cross: a pair that starts low can end high, and another pair can do the opposite. At a crossing point two lines pass through the same $x$ at the same $t$ with different velocities. A field assigns one velocity to each point, so it cannot follow both lines.

The field of the path averages the velocities of all pairs that pass through $x$ at time $t$,

$$
v_t(x) = \mathbb{E}\bigl[X_1 - X_0 \mid X_t = x\bigr] .
$$

This is the marginal field. Its trajectories never cross, because two solutions of $\dot x = v_t(x)$ that meet at some time are the same trajectory.

For the mixture the average has a closed form. Given component $k$, $X_t$ and $X_1 - X_0$ are jointly Gaussian with covariance $t\sigma_k^2 - (1 - t) = s_k \dot s_k$, and conditioning a Gaussian gives a linear function of $x$. Averaging over the components gives

$$
\begin{aligned}
v_t(x) &= \sum_k w_k(x, t)\, u_k(x, t), \\
u_k(x, t) &= \mu_k + \frac{\dot s_k(t)}{s_k(t)}\,(x - t\mu_k),
\end{aligned}
$$

where $\dot s_k = (t\sigma_k^2 - (1 - t))/s_k$ and $w_k(x, t)$ is the responsibility of component $k$ at $x$: its share $\pi_k\, \mathcal{N}(x; t\mu_k, s_k^2) / p_t(x)$ of the density there. The code below implements this field and an integrator for particle trajectories. It then compares the field with the 1D formula, on the grid points where $p_t(x) > 10^{-3}$ as in the dynamic transport section.

```python theme={null}
from scipy.integrate import solve_ivp


def fm_field(t, x):
    """Marginal field of the independent-pair path: responsibility-weighted conditional fields."""
    s, s_dot = _fm_scales(t)
    x = np.asarray(x, dtype=float)[..., None]
    log_w = np.log(PI) + norm.logpdf(x, t * MU, s)
    w = np.exp(log_w - logsumexp(log_w, axis=-1, keepdims=True))
    return np.sum(w * (MU + s_dot / s * (x - t * MU)), axis=-1)


def integrate(field_fn, x0, n_steps, method="euler"):
    """Trajectories of dx/dt = field_fn(t, x) on n_steps equal steps, shape (n_steps + 1, len(x0)).

    method="euler" takes explicit Euler steps; method="rk45" is an accurate adaptive solve
    reported at the same times.
    """
    x0 = np.asarray(x0, dtype=float)
    ts = np.linspace(0, 1, n_steps + 1)
    if method == "euler":
        xs = [x0]
        for k in range(n_steps):
            xs.append(xs[-1] + (ts[k + 1] - ts[k]) * field_fn(ts[k], xs[-1]))
        return np.stack(xs)
    sol = solve_ivp(field_fn, (0, 1), x0, t_eval=ts, rtol=1e-8, atol=1e-8)
    return sol.y.T
```

```python theme={null}
V = np.array([fm_field(t, X_GRID) for t in T_GRID]).T     # shape (len(X_GRID), len(T_GRID))
v_cdf = field_from_cdf(fm_path_cdf, fm_path_density, T_GRID, X_GRID)
m = np.isfinite(v_cdf)
print(f"largest |closed form - 1D formula| where p_t > 1e-3: {np.max(np.abs(v_cdf[m] - V[m])):.1e}")
```

```output theme={null}
largest |closed form - 1D formula| where p_t > 1e-3: 3.4e-05
```

```python theme={null}
T_GRID = np.linspace(0, 1, 201); X_GRID = np.linspace(-6, 7, 2001)
V = np.array([fm_field(t, X_GRID) for t in T_GRID]).T
assert np.all(np.isfinite(V))
v_cdf = field_from_cdf(fm_path_cdf, fm_path_density, T_GRID, X_GRID)
m = np.isfinite(v_cdf)
assert np.max(np.abs(v_cdf[m] - V[m])) < 1e-3
assert np.max(np.abs(v_cdf[m] - V[m])) < 1e-4   # the two computations agree to 1e-4
```

The two computations agree to better than $10^{-4}$. The figure below shows eight pairs drawn independently. On the left each pair moves along its straight conditional path. On the right the same eight starting points follow the marginal field.

```python theme={null}
import torch


def pairs_independent(n, rng=np.random.default_rng(0)):
    """n independent pairs x_0 ~ N(0, 1), x_1 ~ p_1, as float32 tensors."""
    x0 = rng.standard_normal(n)
    comp = rng.choice(len(PI), size=n, p=PI)
    x1 = MU[comp] + SIGMA[comp] * rng.standard_normal(n)
    return torch.tensor(x0, dtype=torch.float32), torch.tensor(x1, dtype=torch.float32)
```

```python theme={null}
a0, a1 = (z.numpy().astype(float) for z in pairs_independent(8, rng=np.random.default_rng(0)))
eight_marginal = integrate(fm_field, a0, 200, method="rk45")
order0, order1 = np.argsort(a0), np.argsort(a1)
assert not np.array_equal(order0, order1)                          # some conditional paths cross
assert np.all(np.diff(eight_marginal[:, order0], axis=1) > 0)      # marginal trajectories never do
```

![Two panels of eight trajectories. Left, straight lines joining eight independent pairs from t = 0 to t = 1; several lines cross each other. Right, the marginal-field trajectories from the same eight starting points; they curve, keep their order and never cross.](https://mintcdn.com/aegeanaiinc/TxnGZcYFzPhEZiXx/aiml-common/lectures/flow-matching/mog-1d/images/cell_14_output_1.png?fit=max\&auto=format\&n=TxnGZcYFzPhEZiXx\&q=85\&s=f11dda969d48127bb65f2c7693bafc5a)

*Left, eight independent pairs joined by straight lines; several lines cross. Right, the trajectories of the marginal field from the same eight starting points. They keep their order and bend instead of crossing.*

## Particles carry the density

The marginal field moves the path: integrating $\dot x = v_t(x)$ from samples of $p_0$ gives samples of $p_t$ at every $t$. The code below integrates 4000 samples of $p_0$ with an accurate solver and compares them at several times with the exact $p_t$. The comparison uses the Kolmogorov-Smirnov (KS) distance, the largest gap between the empirical CDF of the particles and $F_t$.

```python theme={null}
x0 = np.random.default_rng(0).standard_normal(4000)
traj_exact = integrate(fm_field, x0, 200, method="rk45")
ks_floor = stats.kstest(x0, norm.cdf).statistic
SNAPSHOTS = (0.25, 0.5, 0.75, 1.0)
ks_snap = [stats.kstest(traj_exact[round(200 * t)], lambda x, t=t: fm_path_cdf(t, x)).statistic
           for t in SNAPSHOTS]
print("KS distance of the particles to p_t at t = 0.25, 0.5, 0.75, 1:", np.round(ks_snap, 4))
print(f"KS distance of the 4000 samples to p_0: {ks_floor:.4f}")
```

```output theme={null}
KS distance of the particles to p_t at t = 0.25, 0.5, 0.75, 1: [0.0191 0.0191 0.0191 0.0191]
KS distance of the 4000 samples to p_0: 0.0191
```

```python theme={null}
assert max(ks_snap) < 0.03
assert np.allclose(ks_snap, ks_floor, atol=1e-3)   # the flow keeps order, so the KS distance stays put
```

![Top, the standard heatmap of the independent-pair path with 40 white trajectories of the marginal field that stay together, then fan out into three bands. Bottom, four histograms of 4000 particles at t = 0.25, 0.5, 0.75 and 1, each matching the red exact density, which goes from one bump to three.](https://mintcdn.com/aegeanaiinc/TxnGZcYFzPhEZiXx/aiml-common/lectures/flow-matching/mog-1d/images/cell_17_output_1.png?fit=max\&auto=format\&n=TxnGZcYFzPhEZiXx\&q=85\&s=f8e0e4bd16f7d0aa85a04092262f1cb9)

*Top, the standard figure of the independent-pair path with 40 trajectories of the marginal field. Bottom, histograms of all 4000 particles at four times, with the exact $p_t$ in red.*

At every time the KS distance equals the distance of the 4000 starting samples to $p_0$, to three decimal places. The flow keeps the order of the particles and moves $p_0$ to $p_t$, so the particle at quantile level $u$ goes to $F_t^{-1}(u)$, and the empirical CDF keeps its shape. What remains is the sampling error of the initial draw. The same value serves later as the floor for judging learned fields.

## The loss is least squares

For a sample $x_t$ you can compute the velocity $x_1 - x_0$ of its own pair, but not the marginal field $v_t(x_t)$, which averages over all pairs. Flow matching uses a fact about regression. For random variables $X$ and $Y$, the function that minimizes $\mathbb{E}\bigl[(Y - f(X))^2\bigr]$ is the conditional mean $f^\star(x) = \mathbb{E}[Y \mid X = x]$.

Take the input to be $(X_t, t)$, with $t$ uniform on $[0, 1]$, and the target to be $Y = X_1 - X_0$. The minimizer is the marginal field. So you can train a network $v_\theta(x, t)$ on the loss

$$
L(\theta) = \mathbb{E}\Bigl[\bigl(v_\theta(X_t, t) - (X_1 - X_0)\bigr)^2\Bigr],
$$

with per-pair targets you can always compute (Lipman et al., 2022; Yaron et al., 2024). The loss does not go to zero. Its smallest value is the variance of the target around its conditional mean, which is

$$
\begin{aligned}
L_{\min} &= \mathbb{E}\bigl[(X_1 - X_0)^2\bigr] - \int_0^1 \mathbb{E}\bigl[v_t(X_t)^2\bigr]\,dt \\
&= 1 + \mathbb{E}_{p_1}\bigl[X_1^2\bigr] - A ,
\end{aligned}
$$

where $A$ is the kinetic action of the path. The code below defines a network with two hidden layers of 64 units, trains it for 10,000 steps on batches of 4096 independent pairs, and prints the training time.

```python theme={null}
import time
from torch import nn

device = "cuda" if torch.cuda.is_available() else "cpu"
STEPS, BATCH = 10_000, 4096


class VelocityMLP(nn.Module):
    """v_theta(x, t): two hidden layers of 64 units with SiLU, scalar output."""

    def __init__(self, hidden=64):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(2, hidden), nn.SiLU(),
                                 nn.Linear(hidden, hidden), nn.SiLU(),
                                 nn.Linear(hidden, 1))

    def forward(self, x, t):
        return self.net(torch.stack([x, t], dim=-1)).squeeze(-1)


def train_fm(sample_pairs, steps=3000, batch=512, lr=3e-3):
    """Regress v_theta(x_t, t) on x_1 - x_0. Returns the model on the CPU, with .losses."""
    torch.manual_seed(0)
    model = VelocityMLP().to(device)
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, steps)
    gen = torch.Generator().manual_seed(0)
    losses = np.empty(steps)
    for i in range(steps):
        x0, x1 = sample_pairs(batch)
        t = torch.rand(batch, generator=gen)
        x0, x1, t = x0.to(device), x1.to(device), t.to(device)
        loss = ((model((1 - t) * x0 + t * x1, t) - (x1 - x0)) ** 2).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
        sched.step()
        losses[i] = loss.item()
    model = model.cpu().eval()
    model.losses = losses
    return model


def as_field(model):
    """The learned field as a function (t, x) -> v_theta(x, t) on NumPy arrays."""
    def field(t, x):
        x = torch.as_tensor(np.asarray(x), dtype=torch.float32)
        with torch.no_grad():
            return model(x, torch.full_like(x, float(t))).numpy().astype(float)
    return field
```

```python theme={null}
models, train_seconds = {}, {}
start = time.perf_counter()
models["independent"] = train_fm(pairs_independent, steps=STEPS, batch=BATCH)
train_seconds["independent"] = time.perf_counter() - start
A_ind = kinetic_action(fm_path_density, V, T_GRID, X_GRID)
loss_floor = 1 + np.sum(PI * (MU**2 + SIGMA**2)) - A_ind
final_loss = models["independent"].losses[-500:].mean()
print(f"training time on {device}: {train_seconds['independent']:.1f} s")
print(f"mean loss over the last 500 steps: {final_loss:.4f} ; smallest possible loss: {loss_floor:.4f}")
```

```output theme={null}
training time on cuda: 15.1 s
mean loss over the last 500 steps: 2.9557 ; smallest possible loss: 2.9475
```

```python theme={null}
assert abs(final_loss / loss_floor - 1) < 0.02     # the loss settles at its floor, not at zero
```

![Three panels. Left, the training loss falls quickly from about 3.6 and levels off just above a dashed line at the smallest possible loss near 2.95. Middle, learned and exact fields at t = 0.1, 0.5 and 0.9 nearly overlap, with small deviations at the edges. Right, a histogram of samples after 64 Euler steps matching the red three-component target density.](https://mintcdn.com/aegeanaiinc/TxnGZcYFzPhEZiXx/aiml-common/lectures/flow-matching/mog-1d/images/cell_21_output_1.png?fit=max\&auto=format\&n=TxnGZcYFzPhEZiXx\&q=85\&s=d3d60772cb0c9b416468c9ced275c7c9)

*Left, the training loss with the smallest possible loss $L_{\min}$ (dashed). Middle, the learned field and the exact marginal field at three times, drawn where $p_t > 10^{-3}$. Right, the 4000 samples moved by 64 Euler steps of the learned field, against $p_1$ in red.*

The loss levels off within 2% of $L_{\min}$, and the remaining spread is the noise in the per-pair targets. Because the floor is far from zero, the value of the loss does not show how close the learned field is to the exact one. The middle panel compares the two fields directly, and the last subsection measures the error.

## Same endpoint, different path

The marginal field keeps the order of particles, so its endpoint map $x_0 \mapsto x_1$ is increasing. An increasing map that moves $p_0$ to $p_1$ is unique on the line: it is the Monge map $T = F_1^{-1} \circ F_0$ of the [first section](https://aegean.ai/aiml-common/lectures/optimal-transport/maps-and-couplings/index.md). The flow of independent pairs therefore ends where optimal transport ends. The code below checks this on the 4000 samples, for $|x_0| < 2.5$, and computes the kinetic action of the path.

```python theme={null}
end = integrate(fm_field, x0, 1, method="rk45")[-1]
core = np.abs(x0) < 2.5
print(f"largest |endpoint of the flow - T(x_0)| for |x_0| < 2.5: "
      f"{np.max(np.abs(end[core] - monge_map(x0[core]))):.1e}")
print(f"kinetic action, independent pairs: {A_ind:.4f}  ({A_ind / w2:.2f} times W_2^2)")
```

```output theme={null}
largest |endpoint of the flow - T(x_0)| for |x_0| < 2.5: 2.4e-07
kinetic action, independent pairs: 2.4485  (1.62 times W_2^2)
```

```python theme={null}
x0 = np.random.default_rng(0).standard_normal(4000)
end = integrate(fm_field, x0, 1, method="rk45")[-1]
core = np.abs(x0) < 2.5
assert np.max(np.abs(end[core] - monge_map(x0[core]))) < 1e-2

A_ind = kinetic_action(fm_path_density, V, T_GRID, X_GRID)
assert A_ind > 1.01 * w2_squared()
assert A_ind > 1.5 * w2_squared()                       # more than 1.5 W_2^2
assert abs(action_fm_cdf / A_ind - 1) < 1e-3            # the 1D formula gives the same action
```

![Two panels. Left, the endpoint of the flow plotted against the starting point lies exactly on the Monge map curve, which rises steeply near x0 = -0.5 and x0 = 0.8. Right, twelve curved trajectories of the marginal field next to dashed straight geodesic lines from the same starting points to the same endpoints.](https://mintcdn.com/aegeanaiinc/TxnGZcYFzPhEZiXx/aiml-common/lectures/flow-matching/mog-1d/images/cell_24_output_1.png?fit=max\&auto=format\&n=TxnGZcYFzPhEZiXx\&q=85\&s=e2975539e7133b90f0de9c57e42b1a72)

*Left, the endpoint of the flow against the Monge map; the two curves coincide. Right, twelve trajectories of the marginal field and the straight geodesic lines from the same starting points. Each pair of curves shares its endpoints.*

The endpoints agree to about $10^{-7}$, but the paths differ. The flow trajectories bend, while the geodesic moves each point in a straight line at constant speed, and the kinetic action of the independent-pair path is more than one and a half times $W_2^2$. By the Benamou-Brenier formula no path costs less than $W_2^2$, and the independent-pair path is not the minimizer.

The endpoints agree only because the example is 1D, where an increasing transport map is unique. In two or more dimensions the flow of independent pairs moves $p_0$ to $p_1$, but its endpoint map is in general not the optimal transport map.

## Three couplings

Flow matching does not require independent pairs. Any joint distribution of $(x_0, x_1)$ with marginals $p_0$ and $p_1$, any coupling in the sense of the first section, gives a path whose time-1 density is $p_1$. The regression recipe is unchanged. Three couplings are compared here.

1. Independent pairs, as above.
2. The quantile coupling. Draw $U$ uniform on $(0, 1)$ and set $x_0 = F_0^{-1}(U)$ and $x_1 = F_1^{-1}(U)$. This is the optimal coupling of the first section. Its conditional paths are the straight lines of the geodesic, and they never cross, so the marginal field is the geodesic field and the regression target has no noise. This coupling is the reference.
3. Sorted minibatches. Draw a batch of independent pairs, sort the $x_0$ values and the $x_1$ values, and pair them in order. This is the optimal coupling of the two batches, not of the two densities, so it only approximates the quantile coupling, even in 1D. It needs nothing but samples, which is why minibatch optimal transport uses it when the densities are unknown (Tong et al., 2023).

Rectified flow straightens paths another way. It re-pairs each source point with the endpoint of its own learned flow and trains again (Liu et al., 2022).

The code below trains a network on each of the other two couplings with the same settings.

```python theme={null}
def pairs_quantile(n, rng=np.random.default_rng(0)):
    """n pairs (F_0^{-1}(U), F_1^{-1}(U)) with U uniform.

    Sampled as x_1 ~ p_1, U = F_1(x_1), x_0 = F_0^{-1}(U): the same joint distribution,
    without the root finding that F_1^{-1} needs.
    """
    comp = rng.choice(len(PI), size=n, p=PI)
    x1 = MU[comp] + SIGMA[comp] * rng.standard_normal(n)
    x0 = norm.ppf(np.clip(mixture_cdf(x1), U_CLIP, 1 - U_CLIP))
    return torch.tensor(x0, dtype=torch.float32), torch.tensor(x1, dtype=torch.float32)


def pairs_sorted(n, rng=np.random.default_rng(0)):
    """n independent pairs, with the x_0 and x_1 batches each sorted and paired in order."""
    x0, x1 = pairs_independent(n, rng=rng)
    return torch.sort(x0).values, torch.sort(x1).values
```

```python theme={null}
for name, sampler in (("quantile", pairs_quantile), ("sorted", pairs_sorted)):
    start = time.perf_counter()
    models[name] = train_fm(sampler, steps=STEPS, batch=BATCH)
    train_seconds[name] = time.perf_counter() - start
for name, sec in train_seconds.items():
    print(f"training time on {device}, {name} coupling: {sec:.1f} s")
```

```output theme={null}
training time on cuda, independent coupling: 15.1 s
training time on cuda, quantile coupling: 21.4 s
training time on cuda, sorted coupling: 18.6 s
```

Each learned field is compared with its exact field, weighted by the density of the path: the relative error

$$
e = \left(\frac{\sum_{t, x} p_t(x)\,\bigl(v_\theta(x, t) - v_t(x)\bigr)^2}{\sum_{t, x} p_t(x)\, v_t(x)^2}\right)^{1/2}
$$

over the grid. For independent pairs $v_t$ is the closed-form field. For the quantile coupling it is the geodesic field. The sorted coupling has no closed form, and its error is measured against the geodesic field, which it approximates. This error is the training error, except for the sorted coupling, where it also includes the bias of pairing within batches.

The kinetic action of each learned field is the average of $\int_0^1 v_\theta(x_t, t)^2\,dt$ over trajectories from the 4000 samples, computed with an accurate solver. It is compared with $W_2^2$ and with $A_{\text{ref}}$, the average of $(T(x_0) - x_0)^2$ over the same 4000 samples, which is the action of the geodesic estimated on those samples.

The integration error is the root mean square distance between the endpoints after 4, 8, 16 and 64 Euler steps and the endpoints of an accurate solve. Both use the same learned field, so this distance contains no training error. The KS distance of the samples to $p_1$ measures sample quality. It mixes training error, integration error and the sampling error of the 4000 starting points, so it is reported but not used to separate the two kinds of error.

```python theme={null}
def geodesic_field_at(t, x):
    """The geodesic field at one time t, as a function of x."""
    return geodesic_field([t], np.asarray(x, dtype=float))[:, 0]


PATH_DENSITY = {fm_field: fm_path_density, geodesic_field_at: geodesic_density}


def weighted_field_error(model, exact_fn):
    """Relative L2 error of the learned field, weighted by the path density, over T_GRID x X_GRID."""
    field, density_fn = as_field(model), PATH_DENSITY[exact_fn]
    num = den = 0.0
    for t in T_GRID:
        p, v = density_fn(t, X_GRID), exact_fn(t, X_GRID)
        num += np.sum(p * (field(t, X_GRID) - v) ** 2)
        den += np.sum(p * v**2)
    return np.sqrt(num / den)


def action_of(model, n_steps=200):
    """Mean of the integral of v_theta(x_t, t)^2 dt along accurate trajectories from 4000 samples."""
    field = as_field(model)
    starts = np.random.default_rng(0).standard_normal(4000)
    ts = np.linspace(0, 1, n_steps + 1)
    traj = integrate(field, starts, n_steps, method="rk45")
    speed2 = np.stack([field(t, x) ** 2 for t, x in zip(ts, traj)])
    return np.mean(np.trapezoid(speed2, ts, axis=0))
```

```python theme={null}
exact_field = {"independent": fm_field, "quantile": geodesic_field_at, "sorted": geodesic_field_at}
EULER_STEPS = (4, 8, 16, 64)
A_ref = np.mean((monge_map(x0) - x0) ** 2)      # W_2^2 estimated on the same 4000 samples
rows, int_err = {}, {}
for name, model in models.items():
    field = as_field(model)
    accurate_end = integrate(field, x0, 200, method="rk45")[-1]
    euler_ends = [integrate(field, x0, n)[-1] for n in EULER_STEPS]
    ks_euler = [stats.kstest(e, mixture_cdf).statistic for e in euler_ends]
    ks_rk45 = stats.kstest(accurate_end, mixture_cdf).statistic
    a = action_of(model)
    rows[name] = (weighted_field_error(model, exact_field[name]), a / w2, ks_euler, ks_rk45, a / A_ref)
    int_err[name] = [np.sqrt(np.mean((e - accurate_end) ** 2)) for e in euler_ends]

print(f"W_2^2 = {w2:.4f} ; the same average on the 4000 samples, A_ref = {A_ref:.4f}\n")
print("Training error and samples from the accurate solve")
print("  ".join(f"{h:>12}" for h in ["coupling", "field error", "A / W2^2", "A / A_ref", "KS accurate"]))
for name, (err, ratio, _, ks_rk45, ratio_ref) in rows.items():
    print("  ".join(f"{v:>12}" for v in [name, f"{err:.4f}", f"{ratio:.4f}", f"{ratio_ref:.4f}", f"{ks_rk45:.4f}"]))
print("\nIntegration error: RMS of |Euler endpoint - accurate endpoint| for the same learned field")
print("  ".join(f"{h:>12}" for h in ["coupling"] + [f"{n} steps" for n in EULER_STEPS]))
for name, errs in int_err.items():
    print("  ".join(f"{v:>12}" for v in [name] + [f"{e:.4f}" for e in errs]))
print("\nKS distance of the Euler samples to p_1")
print("  ".join(f"{h:>12}" for h in ["coupling"] + [f"{n} steps" for n in EULER_STEPS]))
for name, (_, _, ks_euler, _, _) in rows.items():
    print("  ".join(f"{v:>12}" for v in [name] + [f"{k:.4f}" for k in ks_euler]))
print(f"KS of the exact map on the same 4000 samples: {ks_floor:.4f}")
```

```output theme={null}
W_2^2 = 1.5149 ; the same average on the 4000 samples, A_ref = 1.4956

Training error and samples from the accurate solve
    coupling   field error      A / W2^2     A / A_ref   KS accurate
 independent        0.0470        1.6112        1.6320        0.0226
    quantile        0.0163        0.9844        0.9971        0.0205
      sorted        0.0302        0.9790        0.9916        0.0224

Integration error: RMS of |Euler endpoint - accurate endpoint| for the same learned field
    coupling       4 steps       8 steps      16 steps      64 steps
 independent        0.6374        0.3414        0.1842        0.0492
    quantile        0.0289        0.0165        0.0090        0.0024
      sorted        0.0327        0.0177        0.0093        0.0024

KS distance of the Euler samples to p_1
    coupling       4 steps       8 steps      16 steps      64 steps
 independent        0.1230        0.0643        0.0360        0.0198
    quantile        0.0244        0.0224        0.0211        0.0204
      sorted        0.0281        0.0257        0.0242        0.0229
KS of the exact map on the same 4000 samples: 0.0191
```

```python theme={null}
for name, model in models.items():                 # "independent", "quantile", "sorted"
    rel = weighted_field_error(model, exact_field[name])   # p_t-weighted relative L2
    if name != "sorted":                           # sorted is compared with the geodesic: reported only
        assert rel < 0.10, (name, rel)
    xs = integrate(as_field(model), x0, 200, method="rk45")[-1]
    assert stats.kstest(xs, mixture_cdf).statistic < 0.03, name
    paired = np.max(np.abs(mixture_cdf(xs) - norm.cdf(x0)))
    # The flow keeps order and x0 is shared, so F_1(x_1) should equal F_0(x_0) for every sample.
    assert paired < 0.02, (name, paired)

A = {k: action_of(model) for k, model in models.items()}
assert abs(A["quantile"] / w2_squared() - 1) < 0.03
# Against the same-sample reference A_ref the sampling error cancels, so the bound is tighter.
assert abs(A["quantile"] / A_ref - 1) < 0.01
assert A["quantile"] <= A["sorted"] * 1.02 and A["sorted"] < A["independent"]

# Checks on the comparisons reported below.
assert rows["quantile"][0] < rows["independent"][0]
for name in ("quantile", "sorted"):
    assert rows[name][2][0] < 0.5 * rows["independent"][2][0]
    assert abs(rows[name][1] - 1) < 0.05
assert abs(rows["independent"][1] * w2 / A_ind - 1) < 0.03
assert all(row[3] - ks_floor < 0.01 for row in rows.values())
assert all(row[1] < 1 for name, row in rows.items() if name != "independent") and A_ref < w2
for name in ("quantile", "sorted"):
    assert int_err[name][0] < 0.2 * int_err["independent"][0]
for errs in int_err.values():
    assert all(0.4 < errs[i + 1] / errs[i] < 0.7 for i in range(2))   # 4 -> 8 -> 16 steps
assert all(row[2][-1] - ks_floor < 0.01 for row in rows.values())   # 64 steps: near the floor
ks64 = [row[2][-1] for row in rows.values()]
assert max(ks64) - min(ks64) < 0.01                    # 64 steps: the couplings are within 0.01 of each other
```

![Three standard heatmaps with trajectories from 15 starting points. Left, independent pairs, kinetic action 2.441: white accurate trajectories bend, and the cyan four-step Euler polylines drift away from them. Middle, quantile coupling, action 1.491, and right, sorted minibatches, action 1.483: trajectories are straight lines and the cyan Euler points lie on them.](https://mintcdn.com/aegeanaiinc/TxnGZcYFzPhEZiXx/aiml-common/lectures/flow-matching/mog-1d/images/cell_30_output_1.png?fit=max\&auto=format\&n=TxnGZcYFzPhEZiXx\&q=85\&s=ce223bbc4818cc67c63c435581120b76)

*Trajectories of the three learned fields from 15 starting points: accurate solve in white, four Euler steps in cyan. The background is the exact path density: the independent-pair path on the left, the geodesic in the middle and on the right. The sorted coupling has no closed-form path, so its panel shows the geodesic it approximates. Each title reports the kinetic action of the learned field.*

The first table holds the training error, in the field error column. The quantile field is learned more accurately than the independent one, because its regression target equals the field exactly. The independent target is a noisy sample around the field. With the accurate solver all three fields put their samples on $p_1$, with KS distances close to that of the exact map on the same samples.

The second table holds the integration error. With four steps the endpoints of the independent coupling are far from the accurate ones, while the quantile and sorted errors are less than a fifth as large. Their trajectories are almost straight lines, and Euler steps follow a straight line without error. The independent trajectories bend, and a few straight steps cut the corners. For every coupling the error roughly halves each time the number of steps doubles, as expected of a first-order method. The KS distances in the third table follow the same order at four steps, but at 64 steps they sit near the sampling floor and no longer rank the couplings.

The quantile and sorted fields have kinetic actions within a few percent of $W_2^2$, and the independent field has about the action of its exact path. The quantile and sorted actions print slightly below $W_2^2$, which no path can beat. The cause is the finite sample: the actions are averages over 4000 starting points, and on those points the geodesic itself averages $A_{\text{ref}}$, which is below $W_2^2$. Against $A_{\text{ref}}$ the quantile action agrees to within 1%.

Everything here is 1D. In more dimensions the quantile coupling has no simple form, and sorting is replaced by solving an optimal transport problem on each batch. The batch coupling then approximates the optimal one only roughly, so minibatch paths are straighter than independent ones but in general not straight (Tong et al., 2023).

Key references: (Lipman et al., 2022; Yaron et al., 2024; Tong et al., 2023; Liu et al., 2022)

## References

* Lipman, Y., Chen, R., Ben-Hamu, H., Nickel, M., Le, M. (2022). *Flow Matching for generative modeling*. arXiv \[cs.LG]. [Link](http://arxiv.org/abs/2210.02747)
* Liu, X., Gong, C., Liu, Q. (2022). *Flow straight and fast: Learning to generate and transfer data with rectified flow*. arXiv \[cs.LG]. [DOI](https://doi.org/10.48550/arXiv.2209.03003)
* Tong, A., Fatras, K., Malkin, N., Huguet, G., Zhang, Y., et al. (2023). *Improving and generalizing flow-based generative models with minibatch optimal transport*. arXiv \[cs.LG]. [DOI](https://doi.org/10.48550/arXiv.2302.00482)
* Yaron, L., Marton, H., Peter, H., Neta, S., Matt, L., et al. (2024). *Flow Matching Guide and Code*. arXiv \[cs.LG]. [Link](http://arxiv.org/abs/2412.06264)

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/lectures/flow-matching/mog-1d).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.