Technical Application Note

Technical Application Note
Geometric Vision using Ladybug Cameras
Technical Application Note TAN2012009
Revised January 20, 2014
1.1 Subject
Technical Application Note (TAN2012009): Geometric Vision using Ladybug Cameras
1.2 Applicable Product(s)



Ladybug2
Ladybug3
Ladybug5
1.3 Application Note Description
Effective warping and stitching of the images produced by the camera system's six sensors is
achieved through accurate calibration of the physical location and orientation of the sensors and the
distortion model of the lens. This calibration also enables photogrammetric analysis of image data.
This application note discusses the representation used to describe the physical orientation of all of
the sensors with respect to one another and provides instructions for transforming 2D local points to
3D global points and vice versa.
Coordinate Systems on Ladybug Cameras
Relating Lens Coordinate Systems and the Ladybug Coordinate System
Converting a Pixel Location to a 3D Ray
Converting a Pixel Location to a 3D Ray Corrected for Lens Offset
Converting a 3D Point to a Pixel Location
Calibration Accuracy
Additional Downloads and Support
PGR, the Point Grey Research, Inc. logo, FlyCapture, Digiclops, Dragonfly, Dragonfly Express, Chameleon, Firefly, Flea, Ladybug, Zebra, Scorpion and Triclops are
trademarks or registered trademarks of Point Grey Research, Inc. in Canada and other countries. Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
1.4 Coordinate Systems on Ladybug Cameras
Each lens has its own right-handed 3D coordinate system. As well there is a Ladybug 3D Coordinate
system that is associated with the camera as a whole. This makes a total of seven 3D coordinate
systems on every Ladybug camera. As well, there is a 2D pixel-grid coordinate system for each
sensor.
1.4.1 Lens 3D Coordinate System
Each of the six lenses has its own 3D coordinate system.
 Origin is the optical center of the lens
 Z-axis points out of the sensor towards the scene – i.e. it is the optical axis
 The X- and Y-axes are relative to the pixel grid of the image sensor associated with that lens
o The Y-axis points along the image columns. The positive Y direction is in the
direction of ascending row number. This points down from the point of view of a
normally oriented image
o The X-axis points along the image rows. The positive X direction is in the direction of
ascending column number. This points to the right in a normally oriented image
 This coordinate system is used to represent 3D space from the point-of-view of each
lens/sensor pair. Its units are meters, not pixels.
Figure 1: 3D Sensor Coordinates
1.4.2 Sensor 2D Coordinate System
Each sensor has its own 2D coordinate system.
 The u- and v-axes are the image based 2D image coordinate system for the rectified image
space and are measured in pixels
 The origin of the coordinate system is at the intersection of the optical axis and the rectified
image plane and differs for each sensor
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
2
Point Grey Research TAN2012009


Geometric Vision using Ladybug Cameras
The u-axis points along the rows of the image sensor in the direction of ascending column
number (i.e. to the right)
The v-axis points along the columns in the direction of ascending row number (i.e. down).
Figure 2: 2D Sensor Coordinates
1.4.3 Ladybug Camera Coordinate System
The Ladybug Camera coordinate system is centered within the Ladybug case and is determined by
the position of the 6 lens coordinate systems.
 Origin is the center of the five horizontal camera origins
 Z-axis is parallel to the optical axis of the top lens (lens 5) (*)
 X-axis is parallel to the optical axis of lens 0 (*)
 Y-axis is consistent with a righthanded coordinate system based on the X- and Z-axes
 There may be some variations from LD2 – LD3 – LD5
 (*) Note –due to assembly tolerances the optical axes of lens 5 and lens 0 will typically not be
perfectly perpendicular. The X-axis of the Ladybug Camera coordinate system is adjusted
slightly to ensure that they are perpendicular.
Figure 3: Global Coordinates
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
3
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
1.5 Relating Lens Coordinate Systems and the Ladybug Coordinate
System
The position of each lens coordinate system relative to the Ladybug coordinate system is retrievable
from the Ladybug API. First, use ladybugGetCameraUnitExtrinsics() defined in ladybuggeom.h to
retrieve the 3D translation and Euler angle defined rotation.
/**
* Gets the 6-D extrinsics vector for the specified camera unit.
*
* The 6-D extrinsics vector is in EulerZYX convention (see Craig's
* Introduction to Robotics pg. 45-49). The ordering of the extrinsics
* components are :
*
* - element 0 - Rx - Rotation about X (radians)
* - element 1 - Ry - Rotation about Y (radians)
* - element 2 - Rz - Rotation about Z (radians)
* - element 3 - Tx - Translation along X (meters)
* - element 4 - Ty - Translation along Y (meters)
* - element 5 - Tz - Translation along Z (meters)
*
* By extrinsics, we mean that the corresponding 4x4 transformation matrix
* allows one to map a point in the local camera unit coordinates to that
* of the Ladybug coordinate frame. Where s=sin and c=cos, the format of the
* matrix is given below.
*
* Example:
* To map this to Craig's matrix:
* - Rz = alpha
* - Ry = beta
* - Rx = gamma
*
* - |X'| |((cRz)(cRy)) ((cRz)(sRy)(sRx)-(sRz)(cRx)) ((cRz)(sRy)(cRx)+(sRz)(sRx))
* - |Y'|=|((sRz)(cRy)) ((sRz)(sRy)(sRx)+(cRz)(cRx)) ((sRz)(sRy)(cRx)-(cRz)(sRx))
* - |Z'| |((-sRy))
((cRy)(sRx))
((cRy)(cRx)))
* - |1 | |0
0
0
*
* @param context
- The LadybugContext to access.
* @param uiCamera
- Camera index of interest.
* @param ardEulerZYX - The returned 6-D EulerZYX extrinsics vector.
*
* @return A LadybugError indicating the success of the function.
*/
LADYBUGDLL_API LadybugError
ladybugGetCameraUnitExtrinsics(
LadybugContext context,
unsigned int uiCamera,
double ardEulerZYX[6] );
Tx||X|
Ty||Y|
Tz||Z|
1 ||1|
Note the function comments illustrate how to convert the provided Euler angles (Rx, Ry, Rz) and
translation (Tx, Ty, Tz) into a 4x4 Transform T. Using standard homogeneous transform formulation:
Where
[ ] in the appropriate coordinate frame.
Once this homogeneous transform is obtained, it can be used to calculate the lens coordinate frame
origin in the Ladybug coordinate frame, or the lens coordinate frame axes in the Ladybug coordinate
frame, and vice versa.
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
4
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
1.6 Converting a Pixel Location to a 3D Ray
A common task when using the Ladybug camera for geometric vision is to interpret a pixel location in
a particular image to a 3D ray in the Ladybug Coordinate System. There are a variety of image spaces
from which the pixel may be extracted – for example, spherical, cylindrical, rectified or raw. Users
are encouraged to use raw images for this kind of application. Raw images are the only images which
have not been resampled and consequently should provide the best accuracy when finding or
tracking image features.
To convert a pixel location in a raw image to a 3D ray in the Ladybug Coordinate System the following
steps should be taken:
1. Obtain the focal length for the appropriate camera using
ladybugGetCameraUnitFocalLength()
2. Obtain the image center for camera using ladybugGetCameraUnitImageCenter()
3. Obtain 6D extrinsics vector (Euler angles and translation) for the camera using
ladybugGetCameraUnitExtrinsics()
4. Rectify 2D pixel location using ladybugRectifyPixel()
5. Find the (u,v) pixel coordinate for this rectified image location
6. Transform the rectified 2D pixel location into a 3D ray within the local camera coordinate
system
7. Transform the local 3D ray to a 3D ray in the Labybug Coordinate System
To find the (u,v) pixel location from a rectified (column,row) image position, the image center
information must be taken into account:
Where
the pixel row position and
equals
equals the image center row position.
The rectified image position (u,v) can be transformed to a local 3D ray value by interpreting the
rectified image using the standard pin-hole camera model. Note that the focal length and image
center obtained for the camera is in pixels and is valid only for the rectified image of the specified
camera. To calculate the local 3D ray from the rectified 2D pixel location Z is arbitrary and can be set
to 1. Applying the pin-hole model equations then yields:
]
The vector [
. To convert
this vector to the Ladybug Coordinate System, one simply applies the 3x3 rotational component of
the homogeneous transform shown in section 1.5:
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
5
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
Where R is the upper-left 3x3 submatrix of T. The origin of this vector will be the origin of the local
coordinate system transformed into the Ladybug Camera coordinate system, or
[ ]
1.7 Converting a Pixel Location to a 3D Ray Corrected for Lens
Offset
Mapping a 2D raw pixel to a 3D ray is complicated by the fact that each lens center is offset from the
center of the Ladybug Coordinate System. To get the most accurate results the ray has to have both a
starting point and a direction – not just a direction from the center of the Ladybug Coordinate
System.
Mapping a raw pixel proceeds in two steps:
1. Map the raw pixel to its rectified coordinates – API: ladybugRectifyPixel()
/**
* Maps a distorted (raw) pixel location to its corresponding point in the
* rectified image.
*
* This function must be called after ladybugSetOffScreenImageSize(), which
* sets the resolution of rectified images.
*
* @param context
- The LadybugContext to access.
* @param uiCamera
- Camera index that this image corresponds to.
* @param dDistortedRow - Row coordinate of the distorted (raw) pixel to map.
* @param dDistortedCol - Column coordinate of the distorted (raw) pixel to map.
* @param pdRectifiedRow - Location to return the row coordinate of the same point
*
in the rectified image.
* @param pdRectifiedCol - Location to return the column coordinate of the same
*
point in the rectified image.
*
* @return A LadybugError indicating the success of the function.
*
LADYBUG_OVEREXPOSED is returned if the selected region's average
*
intensity exceeds 254/255 for any channel.
*
* @see
ladybugSetOffScreenImageSize()
*/
LADYBUGDLL_API LadybugError
ladybugRectifyPixel(
LadybugContext context,
unsigned int uiCamera,
double dDistortedRow,
double dDistortedCol,
double* pdRectifiedRow,
double* pdRectifiedCol
2. Map the rectified coordinates to a ray location and direction – API: ladybugRCtoXYZ()
/**
* Projects a 2D point on a specific camera unit into a 3D ray in the Ladybug
* coordinate frame. The ray is defined as its starting point and direction.
* The starting point takes into account the camera unit's offset from the
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
6
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
* center of the ladybug camera which allows accurate projection of the
* ray at distances different from the stitching radius.
*
* @param context
- The LadybugContext to access.
* @param dRectifiedRow - The rectified row of the 2D input point.
* @param dRectifiedCol - The rectified column of the 2D input point.
* @param uiCamera
- The camera unit.
* @param pdLocationX
- The output ray location x component.
* @param pdLocationY
- The output ray location y component.
* @param pdLocationZ
- The output ray location z component.
* @param pdDirectionX - The output ray direction x component.
* @param pdDirectionY - The output ray direction y component.
* @param pdDirectionZ - The output ray direction z component.
*
* @return A LadybugError indicating the success of the function.
*/
LADYBUGDLL_API LadybugError
ladybugRCtoXYZ(
LadybugContext context,
double
dRectifiedRow,
double
dRectifiedCol,
unsigned int
uiCamera,
double*
pdLocationX,
double*
pdLocationY,
double*
pdLocationZ,
double*
pdDirectionX,
double*
pdDirectionY,
double*
pdDirectionZ);
Example code for this mapping type, and other mapping types that don’t rely on the API functionality
as heavily, is available in the ladybugTranslate2dTo3d example, included with the Ladybug SDK.
1.8 Converting a 3D Point to a Pixel Location
Reversing the pixel-to-3D problem described in section 1.6 and 1.7 is slightly complicated by the
requirement to first find which lens a 3D point will project into. Otherwise it is straightforward using
the function ladybugXYZtoRC(). The function header information is provided below.
/**
* Projects a 3D point (with respect to the Ladybug coordinate frames) onto
* the indicated camera unit and returns where it will falls on its rectified
* image.
*
* @param context
- The LadybugContext to access.
* @param dLadybugX
- X coordinate of the point to project.
* @param dLadybugY
- Y coordinate of the point to project.
* @param dLadybugZ
- Z coordinate of the point to project.
* @param uiCamera
- Camera index this image corresponds to.
* @param pdRectifiedRow - The returned rectified row location where the 3D point
*
falls (Will be less than 0 if the point does not
*
project to the rectified image).
* @param pdRectifiedCol - The returned rectified column location where the 3D
point
*
falls (Will be less than 0 if the point does not
*
project to the rectified image).
* @param pdNormalized
- The distance from the rectified pixel to the focal
center
*
normalized by the focal length. Can be set to NULL if
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
7
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
*
the caller is not interested in this value.
*
* @return A LadybugError indicating the success of the function.
*/
LADYBUGDLL_API LadybugError
ladybugXYZtoRC(
LadybugContext context,
double dLadybugX,
double dLadybugY,
double dLadybugZ,
unsigned int uiCamera,
double* pdRectifiedRow,
double* pdRectifiedCol,
double* pdNormalized);
To determine which lenses a 3D point will project into, this function can be used for each of the 6
lens locations (by setting the uiCamera parameter appropriately). If the return code is successful
(LADYBUG_OK) then the 3D point defined by dLadybugX, Y and Z projects into camera uiCamera at
the rectified row,column position provided by pdRectifiedRow and pdRectifiedCol. The location of
the pixel in the raw image can be determined by using ladybugUnrectifypPixel.
1.9 Calibration Accuracy
The average accuracy of a Ladybug5 camera is 4 mm at 10 m distance, or 0.023 degrees.
1.9.1 Local to Ladybug Transform Error
There is an error associated with how well the mathematical model can match the actual position
and orientation of the lens with regards to the Ladybug camera coordinate system. This relates to the
difference between the real physical position and the position calculated during camera calibration.
This is related to:
Relating Lens Coordinate Systems and the Ladybug Coordinate System
Converting a 3D Point to a Pixel Location
1.9.2 Rectification Error
There is an error associated with how well the mathematical model can match the underlying lens
distortion.
This is related to:
Converting a Pixel Location to a 3D Ray
Converting a 3D Point to a Pixel Location
1.9.3 Parallax to Center of Camera Error
The parallax error is an additional source of error, not included in the 4 mm at 10 m accuracy stated
above.
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
8
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
The parallax error is the difference between two rays pointing at the same destination from two
different starting positions. In this case, the difference in position is between the lens center and the
camera coordinate center.
The following diagrams show the angular difference observing the same point in 3D space between
when the lens center is located at the real position and when the lens center is at the center of the
Ladybug coordinate system.
This is related to:
Converting a Pixel Location to a 3D Ray Corrected for Lens Offset
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
9
Point Grey Research TAN2012009
Geometric Vision using Ladybug Cameras
1.10 Additional Downloads and Support
Point Grey Research Inc. endeavors to provide the highest level of technical support possible to our
customers. Most support resources can be accessed through the Support section of our website.
Creating a Customer Login Account
The first step in accessing our technical support resources is to obtain a Customer Login Account. This
requires a valid name and email address. To apply for a Customer Login Account go to the Downloads
page.
Knowledge Base
Our Knowledge Base contains answers to some of the most common support questions. It is
constantly updated, expanded, and refined to ensure that our customers have access to the latest
information.
Product Downloads
Customers with a Customer Login Account can access the latest software and firmware for their
cameras from our Downloads page. We encourage our customers to keep their software and
firmware up-to-date by downloading and installing the latest versions.
Contacting Technical Support
Before contacting Technical Support, have you:
1. Read the product documentation and user manual?
2. Searched the Knowledge Base?
3. Downloaded and installed the latest version of software and/or firmware?
If you have done all the above and still can’t find an answer to your question, contact our Technical
Support team.
Revised 20-Jan-14
Copyright © 2012-2014 Point Grey Research, Inc. All rights reserved.
10

Download Report