Skip to main content
The Intel RealSense D435i is a depth camera. It measures depth by active stereo: two infrared cameras, left and right, view the scene, and an infrared projector adds a pattern that gives the stereo matcher texture to work with on plain surfaces. It also has an RGB camera and an IMU. Intel gives an operating range of about 0.3 to 3 m, a depth field of view of 87° by 58°, and depth images of 1280 by 720 pixels at 30 frames per second (Intel). This page applies the feature-based sensor model to the D435i: how the camera produces the range and bearing of a landmark, and what the IMU contributes. Each of the camera’s sensors has its own coordinate frame, and each is modeled by the pinhole model with its own intrinsics. To combine a depth image with a color image, or to place a detected landmark relative to the robot, you need to know how the frames are oriented and where they sit relative to one another. The first sections cover those frames and the extrinsics between them. The last section uses them to compute range and bearing.

​
Right-handed and left-handed frames

A right-handed frame follows the right-hand rule: point your thumb along the xx-axis and your index finger along the yy-axis, and your middle finger points along the zz-axis. A left-handed frame reverses one axis. Software uses both conventions, so check which one applies before you combine data from two sources. ROS 2 and its viewer RViz2 use right-handed frames. A robot’s body frame follows REP-103:

​
The camera’s frames

In the descriptions below, imagine standing behind the camera and looking where it looks. Left and right, for the infrared sensors and for positions, mean left and right from that point of view. Axes of the RealSense camera frames, seen from behind the camera. The camera frames of a RealSense camera, seen from behind. Every sensor on the camera has two frames:
  • a ROS frame, with xx forward, yy left and zz up, as in the table above
  • an optical frame, with xx right, yy down and zz forward, the convention of the pinhole model
The RealSense ROS wrapper publishes all sensor data in the optical frames, since that is how the sensors measure. Both kinds of frame are right-handed. The wrapper also publishes the transforms that connect them, so you can move data from one frame to the other. The base frame of the camera is camera_link. Its origin is at the left infrared sensor (infra1). The depth frame, the left infrared frame and camera_link coincide, because depth is computed in the left infrared camera’s view. The wrapper publishes a static transform from each sensor’s frame to camera_link, and one from each sensor’s ROS frame to its optical frame. Static transforms of a D435i in RViz2, showing the RGB and right infrared sensor frames. Static transforms for the RGB sensor and the right infrared sensor (infra2) of a D435i, shown in RViz2.

​
Transforms and extrinsics

A ROS transform message gives the pose of the frame child_frame_id in the frame header.frame_id. Applied to a point, it maps coordinates expressed in child_frame_id into coordinates expressed in header.frame_id. Mapping the other way needs the inverse. The ROS documentation calls this a transform from header.frame_id to child_frame_id, a phrase that names the two frames, not the direction in which coordinates move (geometry_msgs/Transform). The camera reports the extrinsics between two sensors the same way. The extrinsics from sensor A to sensor B give the position and orientation of sensor A relative to sensor B. A message has two parts:
  • rotation: nine numbers, a 3×33 \times 3 rotation matrix in column-major order
  • translation: three numbers, a translation in meters
Here are the extrinsics from depth to color on a D435i, as the wrapper publishes them:
The rotation is almost the identity: the two sensors face the same way. The translation places the depth sensor relative to the color sensor. Seen from behind the camera, with the color sensor at the origin, the depth sensor is 0.0148 m (1.48 cm) to its right, along the xx-axis of the optical frame. Positions of the depth and color sensors on the D435i, seen from behind. The depth and color sensors of the D435i, seen from behind the camera.
When the RealSense SDK exports a point cloud to the PLY format, versions 2.19.0 and later convert the points to a left-handed frame. This matches the left-handed default view of MeshLab. Convert the points back before you combine them with data in a right-handed ROS frame.

​
Range and bearing to a landmark

The feature-based model needs, for each landmark, its range rr and bearing ϕ\phi in the robot’s frame. The frames above are what turn a detection in an image into those two numbers.

​
From the depth image

Suppose a detector finds a landmark at pixel (u,v)(u, v) of the color image, for example the center of a fiducial marker. Read the depth at that pixel from the depth image aligned to the color image. The RealSense SDK and the ROS wrapper both produce this alignment, using the depth-to-color extrinsics above. The depth value is ZZ, the coordinate of the point along the optical axis. It is not the distance to the point. With the color camera’s intrinsics fx,fy,cx,cyf_x, f_y, c_x, c_y from the pinhole model, back-projection gives the point in the color camera’s optical frame: pc=[(u−cx) Z/fx(v−cy) Z/fyZ].\mathbf p_c = \begin{bmatrix} (u - c_x)\, Z / f_x \\ (v - c_y)\, Z / f_y \\ Z \end{bmatrix}. The transform from the camera to the robot’s base frame, published as TF from how the camera is mounted, gives the same point as pb=R pc+t\mathbf p_b = R\,\mathbf p_c + \mathbf t. The base frame has xx forward and yy left, so the range and bearing on the ground plane are r=pb,x2+pb,y2,ϕ=atan2⁡(pb,y, pb,x).r = \sqrt{p_{b,x}^2 + p_{b,y}^2}, \qquad \phi = \operatorname{atan2}(p_{b,y},\, p_{b,x}).

​
From the color camera alone

Without depth, a marker of known side length SS still gives the range. If it faces the camera and spans ss pixels in the image, the pinhole model gives Z≈fxS/sZ \approx f_x S / s. Solving for the marker’s full pose from its four corners, as in calibration from known points, also handles a marker seen at an angle. The bearing comes from the horizontal pixel coordinate uu, as before.

​
Noise and visibility

The depth comes from the disparity dd, in pixels, between the left and right infrared images: Z=fb/dZ = f b / d, where ff is the infrared cameras’ focal length in pixels and bb is the distance between them, 50 mm on the D435i. An error σd\sigma_d in the disparity gives a depth error of about σZ≈Z2f b σd.\sigma_Z \approx \frac{Z^2}{f\, b}\, \sigma_d. The range error therefore grows with the square of the distance. A landmark at 2 m has four times the range noise of one at 1 m. The bearing error comes from the error σu\sigma_u in the detected horizontal pixel coordinate. Near the image center it is about σϕ≈σu/fx\sigma_\phi \approx \sigma_u / f_x, and it does not grow with distance. A landmark yields a feature only if the color camera sees it, so the detector can find it, and the depth camera measures it, so it has a range. The color camera’s field of view, 69° by 42°, is narrower than the 87° by 58° depth field (D400 datasheet, Table 3-49), so the color camera sets the limit on bearing. Aligning depth to color does not widen the color image. A pixel column uu runs from 00 to the color image’s width WW, and the bearing formula turns those two edges into bearings. With the color camera’s calibrated intrinsics, the visibility test for a camera that faces forward is 0.3≤r≤3 m,−arctan⁡W−cxfx≤ϕ≤arctan⁡cxfx.0.3 \le r \le 3\ \text{m}, \qquad -\arctan\frac{W - c_x}{f_x} \le \phi \le \arctan\frac{c_x}{f_x} . With the image center in the middle of the image, cx=W/2c_x = W/2, the bearing limit is half the color field of view, about ±35°\pm 35°.

​
The IMU

The IMU has an accelerometer and a gyroscope. It measures how the camera moves, not where the landmarks are, so it gives no range or bearing. It helps the filter in two other ways. The gyroscope measures the turning rate, which can supplement the wheels’ estimate of ω\omega in the motion model. When the robot is not accelerating, the accelerometer measures the direction of gravity, which tells you how the camera is tilted. Two different tilts matter. The first is the camera pitched down relative to the robot. That angle is fixed by the mounting, so it belongs in the rotation RR of the camera-to-base transform above. You measure it once, for example with the accelerometer while the robot stands on a level floor. The second is the robot standing on a slope. Then the base frame itself is tilted relative to gravity, while the map is level. Range and bearing taken in the tilted base frame are wrong: part of the landmark’s height above the floor leaks into its range, and part of its sideways offset leaks into its bearing. The accelerometer gives the roll and pitch of the base, and rotating pb\mathbf p_b by them gives the point in a level frame. Take the range and bearing from that level frame.
Found an error on this page? Report an issue.