> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Bird's-Eye View of the Road

> Mapping a camera image onto the road plane with a homography.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/planning/local-planning/birds-eye-view/index.ipynb)

A planner needs the lane in meters, in the car's own frame: how far the lane center lies to the side, and how sharply the road bends ahead. A forward-facing camera does not give either number directly. Parallel lane lines meet at the horizon, and a meter of road near the car covers many more pixels than a meter 30 m ahead. If the road is flat, a single homography maps every road pixel to its position on the ground. Warping the image with it gives a *bird's-eye view* (BEV), a top-down picture of the road in which lane lines are parallel again and distances can be read in meters.

This section builds that transformation for a simulated camera, finds the lane in the bird's-eye view, and maps the lane back onto the camera image. The last part connects the warped image to the learned bird's-eye grids of current driving stacks.

## The road plane and its homography

The vehicle frame has $x$ pointing forward, $y$ to the left and $z$ up, with its origin on the road below the camera. The camera sits at height $h$ and is tilted down by the pitch angle $\phi$. A point $\mathbf{P}$ in the vehicle frame projects to the pixel

$$
\tilde{\mathbf{p}} \sim K R (\mathbf{P} - \mathbf{C}),
$$

where $K$ holds the [camera intrinsics](https://aegean.ai/aiml-common/lectures/sensor-models/cameras/camera-modeling/index.md), $R$ rotates the vehicle frame into the camera frame, and $\mathbf{C} = (0, 0, h)$ is the camera center. On a flat road every point has $z = 0$, so $\mathbf{P} - \mathbf{C} = X \mathbf{e}_x + Y \mathbf{e}_y - h\, \mathbf{e}_z$, and the projection becomes linear in $(X, Y, 1)$:

$$
\tilde{\mathbf{p}} \sim H \begin{bmatrix} X \\ Y \\ 1 \end{bmatrix},
\qquad
H = K \begin{bmatrix} \mathbf{r}_1 & \mathbf{r}_2 & -h\, \mathbf{r}_3 \end{bmatrix},
$$

where $\mathbf{r}_i$ are the columns of $R$. $H$ is a $3 \times 3$ [homography](https://aegean.ai/aiml-common/lectures/3d-reconstruction/homographies/index.md) from the road plane to the image. Its inverse takes any road pixel back to meters on the ground. Pixels above the horizon have no road point, because their rays never meet the road in front of the camera.

The simulated camera has a 1280 by 720 image, a focal length of 900 px, a height of 1.5 m and a pitch of 3°.

```python theme={null}
# Camera intrinsics: a 1280 by 720 image with a 900 px focal length.
W, H_IMG = 1280, 720
K = np.array([[900.0, 0.0, 640.0], [0.0, 900.0, 360.0], [0.0, 0.0, 1.0]])
CAM_HEIGHT = 1.5  # meters above the road
PITCH = np.deg2rad(3.0)  # camera tilted down


def camera_rotation(pitch):
    """Rows are the camera axes (right, down, forward) in the vehicle frame (x forward, y left, z up)."""
    c, s = np.cos(pitch), np.sin(pitch)
    return np.array([[0.0, -1.0, 0.0], [-s, 0.0, -c], [c, 0.0, -s]])


def ground_homography(K, pitch, height):
    """H maps a road point (X, Y, 1) to its image pixel (u, v, 1), up to scale."""
    R = camera_rotation(pitch)
    return K @ np.column_stack([R[:, 0], R[:, 1], -height * R[:, 2]])


def apply_h(H, pts):
    """Apply a homography to an (N, 2) array of points."""
    p = np.c_[pts, np.ones(len(pts))] @ H.T
    return p[:, :2] / p[:, 2:]


H_true = ground_homography(K, PITCH, CAM_HEIGHT)
print("horizon row:", round(360 - 900 * np.tan(PITCH)), "px")
```

```output theme={null}
horizon row: 313 px
```

The camera image is drawn with the same homography. For each pixel, $H^{-1}$ gives the road point the pixel sees, and a function of $(X, Y)$ paints asphalt, grass or white paint there. The lane is 3.7 m wide. Its left edge is a solid line, and its right edge is a dashed line with 3 m dashes and 9 m gaps. A second lane lies to the right. The car drives 0.3 m to the right of the lane center, and the road bends left with a radius of 150 m. A second image of a straight road is used below for calibration.

```python theme={null}
LANE_WIDTH, LINE_WIDTH = 3.7, 0.15  # meters
DASH, GAP = 3.0, 9.0  # dashed line: 3 m painted, 9 m gap


def lane_center(X, radius, offset):
    """Lateral position of the lane center at distance X; the car is offset to the right of it."""
    curve = 0.0 if np.isinf(radius) else X**2 / (2 * radius)
    return offset + curve


def road_color(X, Y, radius, offset):
    """RGB of the road surface at ground points (X, Y)."""
    yc = lane_center(X, radius, offset)
    d = Y - yc  # lateral position relative to the lane center
    rng = np.random.default_rng(0)
    noise = rng.normal(0, 6, 4096)[(np.floor(X * 10).astype(int) * 7919 + np.floor(Y * 10).astype(int)) % 4096]
    rgb = np.empty(X.shape + (3,))
    rgb[:] = np.array([95, 140, 70])  # grass
    asphalt = (d < LANE_WIDTH / 2 + 1.0) & (d > -3 * LANE_WIDTH / 2 - 1.0)  # two lanes and 1 m shoulders
    rgb[asphalt] = 90 + noise[asphalt, None]
    solid = np.abs(d - LANE_WIDTH / 2) < LINE_WIDTH / 2  # left edge of our lane
    dashed = (np.abs(d + LANE_WIDTH / 2) < LINE_WIDTH / 2) & ((X % (DASH + GAP)) < DASH)  # right edge
    outer = np.abs(d + 3 * LANE_WIDTH / 2) < LINE_WIDTH / 2  # right edge of the next lane
    rgb[solid | dashed | outer] = 235
    return rgb


def render(H, radius, offset, supersample=2):
    """Render the camera image by casting every pixel onto the road through the inverse homography."""
    s = supersample
    u, v = np.meshgrid((np.arange(W * s) + 0.5) / s, (np.arange(H_IMG * s) + 0.5) / s)
    g = np.stack([u.ravel(), v.ravel(), np.ones(u.size)]) 
    g = np.linalg.inv(H) @ g
    ground = g[2] > 1e-9  # rays that hit the road in front of the camera
    X, Y = g[0] / np.where(ground, g[2], 1), g[1] / np.where(ground, g[2], 1)
    ground &= X < 300
    img = np.empty((u.size, 3))
    img[:] = np.array([185, 210, 235])  # sky
    img[ground] = road_color(X[ground], Y[ground], radius, offset)
    img = img.reshape(H_IMG, s, W, s, 3).mean(axis=(1, 3))
    return np.clip(img, 0, 255).astype(np.uint8)


RADIUS, OFFSET = 150.0, 0.3  # a left curve of 150 m radius; the lane center is 0.3 m left of the car
curved = render(H_true, RADIUS, OFFSET)
straight = render(H_true, np.inf, 0.0)
```

![Simulated camera image of a two-lane road bending left, with a solid line on the left of the lane, a dashed line on the right, grass beside the road and sky above the horizon.](https://mintcdn.com/aegeanaiinc/49cW2M54qfwz3vnn/aiml-common/lectures/planning/local-planning/birds-eye-view/images/cell_4_output_1.svg?fit=max\&auto=format\&n=49cW2M54qfwz3vnn\&q=85\&s=86e5a54d1c00470562f9269edab3e5a8)

The lane lines meet near the horizon at row 313, and the same meter of road shrinks with distance, so neither the curvature nor the car's offset in the lane can be read from the image.

## Calibrating the homography from four points

In the simulation $H$ is known. The height and pitch of a real camera are not known to the millimeter, so $H$ is usually calibrated on the road itself. On a straight, flat stretch, the centers of the two lane lines are 3.7 m apart, and points along them can be marked at known distances ahead. Four such points determine a homography. The [direct linear transform](https://aegean.ai/aiml-common/lectures/3d-reconstruction/homographies/index.md) solves for it from the eight linear equations the four pairs give. Here the points sit on both lines at 8 m and 30 m, and each is marked with half a pixel of error.

```python theme={null}
def dlt(src, dst):
    """Homography with dst ~ H src from four point pairs (h33 fixed to 1)."""
    A, b = [], []
    for (x, y), (u, v) in zip(src, dst):
        A.append([x, y, 1, 0, 0, 0, -u * x, -u * y]); b.append(u)
        A.append([0, 0, 0, x, y, 1, -v * x, -v * y]); b.append(v)
    return np.append(np.linalg.solve(np.array(A), np.array(b)), 1.0).reshape(3, 3)


# Four road points with known positions: the centers of both lane lines, 8 m and 30 m ahead.
road_pts = np.array([[8.0, 1.85], [30.0, 1.85], [30.0, -1.85], [8.0, -1.85]])
rng = np.random.default_rng(1)
clicked = apply_h(H_true, road_pts) + rng.normal(0, 0.5, (4, 2))  # marked by hand, half-pixel error
H_est = dlt(road_pts, clicked)
print("image points (px):\n", np.round(clicked, 1))
for X in (10.0, 20.0, 30.0, 40.0):
    pt = np.array([[X, 1.85]])
    err = np.linalg.norm(apply_h(np.linalg.inv(H_est), apply_h(H_true, pt)) - pt)
    print(f"position error of a lane-line point {X:.0f} m ahead: {100 * err:.0f} cm")
```

```output theme={null}
image points (px):
 [[433.8 480.8]
 [584.7 357.2]
 [695.9 358.1]
 [846.1 480.7]]
position error of a lane-line point 10 m ahead: 0 cm
position error of a lane-line point 20 m ahead: 17 cm
position error of a lane-line point 30 m ahead: 43 cm
position error of a lane-line point 40 m ahead: 77 cm
```

The error grows with distance. One pixel row covers about $X^2 / (f h)$ meters of road at distance $X$: 5 cm at 8 m, 67 cm at 30 m and 1.2 m at 40 m. Half a pixel of error at the far points therefore moves the far end of the map by tens of centimeters, while the near end stays exact. This is one reason the bird's-eye view below stops at 36 m.

## Warping to the bird's-eye view

The bird's-eye view is a grid on the road, here from 6 to 36 m ahead and 6 m to each side, at 5 cm per pixel. A second matrix $A$ maps a grid pixel to its road point, so $HA$ maps a grid pixel straight to the camera pixel that sees it. The warp fills each grid pixel by sampling the camera image at that position with bilinear interpolation. This is inverse warping: every output pixel gets exactly one value, and the result has no holes.

```python theme={null}
BEV_RES = 0.05  # meters per bird's-eye-view pixel
X_NEAR, X_FAR, Y_SPAN = 6.0, 36.0, 6.0  # the grid covers 6 to 36 m ahead and 6 m to each side
BEV_ROWS, BEV_COLS = int((X_FAR - X_NEAR) / BEV_RES), int(2 * Y_SPAN / BEV_RES)
# A maps a bird's-eye pixel (column, row, 1) to a road point (X, Y, 1): far is up, left is left.
A = np.array([[0.0, -BEV_RES, X_FAR], [-BEV_RES, 0.0, Y_SPAN], [0.0, 0.0, 1.0]])


def bilinear(img, u, v):
    """Sample img at real-valued pixel positions; black outside the image."""
    u0, v0 = np.floor(u - 0.5).astype(int), np.floor(v - 0.5).astype(int)
    fu, fv = u - 0.5 - u0, v - 0.5 - v0
    out = np.zeros(u.shape + (img.shape[2],))
    for du, dv, w in ((0, 0, (1 - fu) * (1 - fv)), (1, 0, fu * (1 - fv)), (0, 1, (1 - fu) * fv), (1, 1, fu * fv)):
        uu, vv = u0 + du, v0 + dv
        ok = (uu >= 0) & (uu < img.shape[1]) & (vv >= 0) & (vv < img.shape[0])
        out[ok] += w[ok, None] * img[vv[ok], uu[ok]]
    return out


def birds_eye(img, H):
    """Warp a camera image to the bird's-eye grid: one homography, H A, per pixel."""
    cols, rows = np.meshgrid(np.arange(BEV_COLS) + 0.5, np.arange(BEV_ROWS) + 0.5)
    uv = apply_h(H @ A, np.c_[cols.ravel(), rows.ravel()])
    return bilinear(img, uv[:, 0], uv[:, 1]).reshape(BEV_ROWS, BEV_COLS, 3).astype(np.uint8)


bev_straight = birds_eye(straight, H_est)
bev_curved = birds_eye(curved, H_est)
```

![Left: camera image of a straight road with four marked points on the two lane lines forming a trapezoid. Right: the bird's-eye view, in which the lane lines are parallel and the four points form a rectangle.](https://mintcdn.com/aegeanaiinc/49cW2M54qfwz3vnn/aiml-common/lectures/planning/local-planning/birds-eye-view/images/cell_7_output_1.svg?fit=max\&auto=format\&n=49cW2M54qfwz3vnn\&q=85\&s=3c8364e6fc6cc0993b63a6f94b5ec01a)

On the straight road the lane lines are parallel in the bird's-eye view, and the four marked points form a rectangle 3.7 m wide. The near rows are sharp. The far rows are blurred along the direction of travel, because each camera pixel there is stretched over many grid pixels. The black triangles at the bottom are road outside the camera's field of view.

## Finding the lane

In the bird's-eye view a lane line is a curve $Y(X)$ in meters. The code thresholds the bright paint and builds a histogram of its lateral positions over the 10 m nearest the car. The peak within 3 m on each side gives the starting position of each line. Sliding windows then follow each line forward. Each window collects the paint pixels near the line, recenters on them, and steps ahead along the line's direction, which also carries it across the gaps of the dashed line. A quadratic $Y = aX^2 + bX + c$ fitted to the collected pixels gives the radius of curvature at the car ($X = 0$),

$$
\rho = \frac{(1 + b^2)^{3/2}}{|2a|},
$$

and the lane center at the car is the mean of the two fitted values of $c$.

```python theme={null}
def lane_pixels(bev, threshold=180):
    """Road coordinates (X, Y) of the bright pixels in a bird's-eye image."""
    rows, cols = np.nonzero(bev.mean(axis=2) > threshold)
    XY = apply_h(A, np.c_[cols + 0.5, rows + 0.5])
    return XY[:, 0], XY[:, 1]


def sliding_window(X, Y, y_start, n_windows=15, half_width=0.5):
    """Follow one line from the near edge of the grid; return its pixels and the window centers."""
    edges = np.linspace(X_NEAR, X_FAR, n_windows + 1)
    keep = np.zeros(X.shape, bool)
    centers, found, y_c = [], [], y_start
    for lo, hi in zip(edges[:-1], edges[1:]):
        inside = (X >= lo) & (X < hi) & (np.abs(Y - y_c) < half_width)
        keep |= inside
        centers.append((lo, hi, y_c))
        if inside.sum() > 20:  # recenter on the pixels in this window
            found.append((X[inside].mean(), Y[inside].mean()))
        step = hi - lo
        if len(found) >= 2:  # continue along the line's direction, also across a dash gap
            (x1, y1), (x2, y2) = found[-2], found[-1]
            y_c = y2 + (y2 - y1) / (x2 - x1) * (hi + step / 2 - x2)
        elif found:
            y_c = found[-1][1]
    return keep, centers


def curvature_radius(p, X=0.0):
    a, b = p[0], p[1]
    return (1 + (2 * a * X + b) ** 2) ** 1.5 / abs(2 * a)


def find_lane(bev):
    """Fit Y = a X^2 + b X + c to each lane line; return fits, windows, radius and lane center at the car."""
    X_px, Y_px = lane_pixels(bev)
    near = X_px < X_NEAR + 10  # columns of bright pixels in the 10 m nearest the car
    hist, bins = np.histogram(Y_px[near], bins=np.arange(-3.0, 3.0 + 0.1, 0.1))
    mid = 0.5 * (bins[:-1] + bins[1:])
    starts = {"left": mid[mid > 0][np.argmax(hist[mid > 0])],  # the line within 3 m on the left
              "right": mid[mid < 0][np.argmax(hist[mid < 0])]}  # and on the right
    fits, windows = {}, {}
    for side, y0 in starts.items():
        keep, windows[side] = sliding_window(X_px, Y_px, y0)
        fits[side] = np.polyfit(X_px[keep], Y_px[keep], 2)
    radius = np.mean([curvature_radius(p) for p in fits.values()])
    center = np.mean([np.polyval(p, 0.0) for p in fits.values()])
    return fits, windows, radius, center


fits, windows, radius_est, center_est = find_lane(bev_curved)
print(f"radius of curvature at the car: {radius_est:.0f} m (true {RADIUS:.0f} m)")
print(f"lane center: {center_est:.2f} m to the left of the car (true {OFFSET:.2f} m)")
_, _, radius_exact, center_exact = find_lane(birds_eye(curved, H_true))
print(f"with the exact H instead: {radius_exact:.0f} m and {center_exact:.2f} m")
```

```output theme={null}
radius of curvature at the car: 141 m (true 150 m)
lane center: 0.36 m to the left of the car (true 0.30 m)
with the exact H instead: 152 m and 0.28 m
```

![Bird's-eye view of the curved road with the sliding windows that follow each lane line and the quadratic fit through each line.](https://mintcdn.com/aegeanaiinc/49cW2M54qfwz3vnn/aiml-common/lectures/planning/local-planning/birds-eye-view/images/cell_9_output_1.svg?fit=max\&auto=format\&n=49cW2M54qfwz3vnn\&q=85\&s=3401d52797aa129dc772b53cdbd59cd8)

The estimates are 141 m against the true 150 m, and 0.36 m against the true 0.30 m. With the exact homography the same code gives 152 m and 0.28 m, so most of the error comes from the half-pixel marking error in the calibration, which stretches the far end of the map.

## Back to the camera image

To show the result in the camera image, each camera pixel is sent to the road through $H^{-1}$. The pixel is painted when its road point lies between the two fitted lines and inside the grid. Only the inverse of the same homography is needed.

```python theme={null}
def lane_overlay(img, H, fits, color=(0, 158, 115), alpha=0.35):
    """Paint the area between the two fitted lines back onto the camera image."""
    u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H_IMG) + 0.5)
    g = np.linalg.inv(H) @ np.stack([u.ravel(), v.ravel(), np.ones(u.size)])
    ok = g[2] > 1e-9
    X = np.where(ok, g[0] / np.where(ok, g[2], 1), -1)
    Y = np.where(ok, g[1] / np.where(ok, g[2], 1), 0)
    inside = ok & (X > X_NEAR) & (X < X_FAR) & (Y < np.polyval(fits["left"], X)) & (Y > np.polyval(fits["right"], X))
    out = img.reshape(-1, 3).astype(float)
    out[inside] = (1 - alpha) * out[inside] + alpha * np.array(color)
    return out.reshape(img.shape).astype(np.uint8)


overlay = lane_overlay(curved, H_est, fits)
```

![The camera image of the curve with the detected lane shaded green between the fitted lines, and the estimated radius and lane-center offset printed in a corner.](https://mintcdn.com/aegeanaiinc/49cW2M54qfwz3vnn/aiml-common/lectures/planning/local-planning/birds-eye-view/images/cell_11_output_1.svg?fit=max\&auto=format\&n=49cW2M54qfwz3vnn\&q=85\&s=2b93b3c3e80e0fdf93cb01867adfd1ac)

## When the road plane moves

The homography is valid only for the road plane it was calibrated on. When the car brakes or drives over a bump it pitches, and the camera's real pitch no longer matches $H$. Below, the car is pitched 1° nose up, and the image is warped with the calibrated $H$.

```python theme={null}
# The car pitches up by 1 degree, for example as braking ends; the warp still uses the calibrated H.
H_pitched = ground_homography(K, PITCH - np.deg2rad(1.0), CAM_HEIGHT)
pitched = render(H_pitched, RADIUS, OFFSET)
bev_pitched = birds_eye(pitched, H_est)
# Where the calibrated warp puts points of the two lane lines that are really 10, 20 and 30 m ahead.
for X in (10.0, 20.0, 30.0):
    yc = lane_center(X, RADIUS, OFFSET)
    true_pts = np.array([[X, yc + LANE_WIDTH / 2], [X, yc - LANE_WIDTH / 2]])
    seen = apply_h(np.linalg.inv(H_est), apply_h(H_pitched, true_pts))
    print(f"{X:.0f} m ahead appears at {seen[:, 0].mean():.1f} m, lane width {seen[0, 1] - seen[1, 1]:.2f} m")
```

```output theme={null}
10 m ahead appears at 9.0 m, lane width 3.30 m
20 m ahead appears at 16.1 m, lane width 2.94 m
30 m ahead appears at 21.9 m, lane width 2.64 m
```

![Two bird's-eye views of the same curve. With the calibrated pitch the lane has constant width; with the car pitched up by one degree the lane lines converge with distance.](https://mintcdn.com/aegeanaiinc/49cW2M54qfwz3vnn/aiml-common/lectures/planning/local-planning/birds-eye-view/images/cell_13_output_1.svg?fit=max\&auto=format\&n=49cW2M54qfwz3vnn\&q=85\&s=0b8464c8b2b150e2bd65cc6c8416968d)

A point 10 m ahead lands at 9.0 m, one 20 m ahead at 16.1 m, and one 30 m ahead at 21.9 m. The lane narrows from 3.7 m to 3.30, 2.94 and 2.64 m, so the lines in the bird's-eye view converge and the curve looks tighter. A curvature estimate from this frame would be wrong. A hill or a banked road breaks the flat-road assumption in the same way. Correcting for it needs the pitch in every frame, from an IMU for example, or a road model that is not a single plane.

## From a warped image to a bird's-eye grid

The homography maps only the road plane. Anything with height, such as a car, a pedestrian or a curb, is smeared away from the camera in the warped image, because the homography treats every pixel as a point on the road. Driving stacks therefore treat the bird's-eye view as a grid to fill rather than an image to warp. Lidar returns are projected into the grid directly. For cameras, networks learn to transform the images of several cameras into one bird's-eye grid around the car and label each cell, for example as drivable road or as an obstacle. Tesla's occupancy networks predict a 3D grid of occupied voxels from eight cameras (Elluswamy, 2022). These grids are [occupancy grids](https://aegean.ai/aiml-common/lectures/rse/occupancy-mapping/index.md#occupancy-grids-in-practice) filled by a learned inverse sensor model, and the planner uses them the same way.

## References

* Elluswamy, A. (2022). Keynote, Workshop on Autonomous Driving, CVPR 2022. [Video](https://www.youtube.com/watch?v=jPCV4GKX9Dw).
* Monogioudis, P. (2017). [Advanced lane finding](https://github.com/pantelis/Advanced-Lane-Lines). Perspective transform to a bird's-eye view, sliding-window lane search and curvature estimation.

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/lectures/planning/local-planning/birds-eye-view).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.