Pinhole Camera Model

Camera Models

Camera models describe how a camera projects points in the 3D world onto a 2D image plane. Each model makes different assumptions. Common camera models include:

The Pinhole Camera Model: The pinhole model is the simplest camera model. It assumes that light rays pass through a single point, called the pinhole, before reaching the image plane. Its parameters include the focal length and the position of the pinhole.
The Perspective Camera Model: The perspective model extends the pinhole model to include lens distortion and other optical effects. It is used in computer vision and graphics to model camera behavior.
The Orthographic Camera Model: In the orthographic model, parallel lines in the 3D world remain parallel in the 2D image. This property is useful for technical drawings and architectural visualizations that require accurate measurements.
The Spherical Camera Model: The spherical model uses a spherical image sensor to capture a 360-degree view of a scene. It is used in virtual reality and panoramic photography.
The Omnidirectional Camera Model: The omnidirectional model uses multiple lenses or a fisheye lens to capture a wide field of view (FOV). It is used in robotics and surveillance.

Each model defines equations and parameters for projecting 3D points onto the 2D image plane. These models are used for image rectification, 3D reconstruction, and camera calibration. This section focuses on the pinhole camera model.

Pinhole camera model. The image is formed on the image plane by light rays passing through a small aperture (the pinhole) at the center of projection. The image is inverted and smaller than the object.

Camera Model Fundamentals

Pinhole Camera Model

The functions in this section use the pinhole camera model. A perspective transformation projects a 3D scene point

P_w

onto the image plane at pixel

p

. The variables

P_w

and

p

are represented in homogeneous coordinates as 3D and 2D vectors, respectively. The pinhole camera model gives the following distortion-free projective transformation:

\lambda \; p = K \begin{bmatrix} R|t \end{bmatrix} P_w

where:

$P_w$ is a 3D point expressed with respect to the world coordinate system
$p$ is a 2D pixel in the image plane
$K$ is the camera intrinsic matrix
$R$ and $t$ are the rotation and translation that describe the change of coordinates from world to camera coordinate systems
$\lambda$ is the projective transformation’s arbitrary scaling

Camera Intrinsic Matrix

The camera intrinsic matrix

K

maps 3D points in the camera coordinate system to 2D pixel coordinates, up to a scale factor:

\lambda\, p = K P_c, \qquad p = \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}, \quad \lambda = Z_c

Dividing by

\lambda

is the perspective division that turns

K P_c

into pixel coordinates. The matrix

K

contains the focal lengths

f_x

and

f_y

, expressed in pixels, and the principal point

(c_x, c_y)

, which is usually near the image center:

K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}

Therefore:

\lambda \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} X_c \\ Y_c \\ Z_c \end{bmatrix}

Coordinate Transformations

The joint rotation-translation matrix

[R|t]

factors into a 3-by-4 projection and a 4-by-4 homogeneous transformation (Hartley & Zisserman, 2004; Szeliski, n.d.):

[R|t] = \begin{bmatrix} I & 0 \end{bmatrix} \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix}

The homogeneous transformation, built from the extrinsic parameters, maps a world point to camera coordinates

(X_c, Y_c, Z_c)

. The projection

\begin{bmatrix} I & 0 \end{bmatrix}

keeps these three coordinates and drops the homogeneous one. Read as a homogeneous 2D point, the result gives the normalized camera coordinates

x' = X_c / Z_c

and

y' = Y_c / Z_c

on the image plane. The extrinsic parameters

R

and

t

define the homogeneous transformation from the world coordinate system

w

to the camera coordinate system

c

P_c = \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix} P_w

The complete transformation is:

\lambda \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_x \\ r_{21} & r_{22} & r_{23} & t_y \\ r_{31} & r_{32} & r_{33} & t_z \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}

Z_c \neq 0

, this is equivalent to:

\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} f_x X_c/Z_c + c_x \\ f_y Y_c/Z_c + c_y \end{bmatrix}

Lens Distortion Model

Real lenses introduce radial and tangential distortion.

Distortion examples: barrel, pincushion, and tangential distortions. The extended camera model includes this distortion:

\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} f_x x'' + c_x \\ f_y y'' + c_y \end{bmatrix}

where:

\begin{bmatrix} x'' \\ y'' \end{bmatrix} = \begin{bmatrix} x' \frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + 2 p_1 x' y' + p_2(r^2 + 2 x'^2) + s_1 r^2 + s_2 r^4 \\ y' \frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + p_1 (r^2 + 2 y'^2) + 2 p_2 x' y' + s_3 r^2 + s_4 r^4 \end{bmatrix}

with

r^2 = x'^2 + y'^2

and

\begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} X_c/Z_c \\ Y_c/Z_c \end{bmatrix}

Z_c \neq 0

. Distortion parameters:

Radial coefficients: $k_1$ , $k_2$ , $k_3$ , $k_4$ , $k_5$ , $k_6$
Tangential coefficients: $p_1$ , $p_2$
Thin prism coefficients: $s_1$ , $s_2$ , $s_3$ , $s_4$

The distortion coefficients are passed as:

(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \tau_x, \tau_y]]]])

Types of distortion: The radial part of the model scales each point by the full radial factor

\rho(r) = \frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6},

so the type of radial distortion depends on how the whole ratio changes with

r

, not on the numerator alone:

Barrel distortion: $\rho(r)$ decreases as $r$ grows, so points far from the center move inward.
Pincushion distortion: $\rho(r)$ increases as $r$ grows, so points far from the center move outward.

Right-handed and Left-handed Coordinate Systems

Right-handed and left-handed coordinate systems use different conventions for orienting axes in 3D space. Applications may use either convention, so you must know which one applies.

ROS2

ROS2 RViz2 uses a right-handed coordinate system. The right-hand rule determines the axis directions. Point your thumb along the X-axis and your index finger along the Y-axis. Your middle finger then points along the Z-axis.

Axis	Direction	Color
X	Forward	🔴 Red
Y	Left	🟢 Green
Z	Up	🔵 Blue

Sensor Coordinate Systems

Each sensor uses a coordinate system documented by the vendor. It may be right-handed or left-handed. RealSense cameras use a right-handed coordinate system.

Point of view:
- Imagine that you are standing behind the camera and looking forward.
- Use this point of view when describing coordinates, left and right infrared sensors, and sensor positions.

ROS2 Coordinate System: (X: Forward, Y: Left, Z: Up)
Camera Optical Coordinate System: (X: Right, Y: Down, Z: Forward)
References: REP-0103, REP-0105
All data published by the RealSense wrapper is optical data taken directly from the camera sensors.
Static and dynamic TF topics publish the optical and ROS coordinate systems. You can use these transforms to move from one coordinate system to the other.

TF from Coordinate A to Coordinate B

A TF message gives the pose of the coordinate frame child_frame_id in the coordinate frame “header.frame_id”. Applied to a point, it maps coordinates in child_frame_id into coordinates in header.frame_id; mapping the other way needs its inverse. The ROS documentation describes this as a transform from header.frame_id to child_frame_id, which refers to the frames, not to the direction in which coordinates are mapped. See the reference.
In RealSense cameras, the origin (0,0,0) is the position of the left infrared sensor (infra1). This origin defines the “camera_link” frame.
The depth, left infrared, and “camera_link” coordinate frames coincide.
The wrapper provides static TFs from each sensor coordinate frame to the camera base (camera_link).
It also provides TFs from each sensor ROS coordinate frame to the corresponding optical coordinate frame.
The following image shows the static TFs for the RGB sensor and the Infra2 (right infrared) sensor of a D435i module in RViz2:

Extrinsics from Sensor A to Sensor B

The extrinsic transform from sensor A to sensor B gives the position and orientation of sensor A relative to sensor B.
If B is the origin (0,0,0), then Extrinsics(A->B) describes the position of sensor A relative to sensor B.
For example, consider depth_to_color for the D435i:
- Viewed from behind the D435i, the extrinsic transform from depth to color specifies the position of the depth sensor relative to the color sensor.
- Consider only the X coordinates in the optical coordinate system, viewed from behind. If the COLOR (RGB) sensor is at (0,0,0), the DEPTH sensor is 0.0148 m (1.48 cm) to its right.

administrator@perclnx466 ~/ros2_humble $ ros2 topic echo /camera/camera/extrinsics/depth_to_color
rotation:
- 0.9999583959579468
- 0.008895332925021648
- -0.0020127370953559875
- -0.008895229548215866
- 0.9999604225158691
- 6.045500049367547e-05
- 0.0020131953060626984
- -4.254872692399658e-05
- 0.9999979734420776
translation:
- 0.01485931035131216
- 0.0010161789832636714
- 0.0005317096947692335
---

An extrinsic message has two parts:
- float64[9] rotation (column-major 3x3 rotation matrix)
- float64[3] translation (three-element translation vector in meters)

When exporting vertices to PLY format, RealSense SDK versions 2.19.0 and later convert the points to a left-handed coordinate system. PLY is a common 3D file format. The conversion provides compatibility with the default left-handed viewpoint in MeshLab.

References

Hartley, R., Zisserman, A. (2004). Multiple View Geometry in Computer Vision.
Szeliski, R. (n.d.). Computer Vision: Algorithms and Applications.

Found an error on this page? Report an issue.

Perception

Camera Sensor Models

Beam Sensor Models

Image Processing

CNNs

Scene Understanding

Faster RCNN

Mask RCNN Lab

YOLO Lab

UNet Lab

Pinhole Camera Model

Camera Models