1. CV: 3D sensing and calibration
Coordinate system changes;
perspective transformation;
Stereo and structured light
MSU CSE 803 Fall 2008 Stockman

2. roadmap using multiple cameras using structured light projector
3D transformations
general perspective transformation
justification of 3x4 camera model
MSU CSE 803 Fall 2008 Stockman

3. Four Coordinate frames
W: world,
C,D: cameras,
M: object model
Need to relate all to each other.
MSU CSE 803 Fall 2008 Stockman

4. Can we recognize? Is there some object M
That can be placed in some location
That will create the two images that are observed?
Discover/compute what object and what pose
MSU CSE 803 Fall 2008 Stockman

5. Need to relate frames to compute
relate camera to world using rotations and translations
project world point into real image using projection
scale image point in real image plane to get pixel array coordinates
MSU CSE 803 Fall 2008 Stockman

6. Stereo configuration
2 corresponding image points enable the intersection of 2 rays in W
MSU CSE 803 Fall 2008 Stockman

7. Stereo computation
MSU CSE 803 Fall 2008 Stockman

8. Math for stereo computations
need to calibrate both cameras to W so that rays in x,y,z reference same space
need to have corresponding points
find point of closest approach of the two rays
(rays are too far apart  point correspondence error or crude calibration)
MSU CSE 803 Fall 2008 Stockman

9. Replace camera with projector
Can calibrate a projector to W easily. Correspondence now means identifying marks.
MSU CSE 803 Fall 2008 Stockman

10. Advantages/disadvantages of structured light
MSU CSE 803 Fall 2008 Stockman

11. Grid projected on objects
All grid intersects are integral
MSU CSE 803 Fall 2008 Stockman

12. Computing surface normals
Surface normals have been computed and then added to the image (augmented reality)
MSU CSE 803 Fall 2008 Stockman

13. Relating coordinate frames
need to relate camera frame to world
need to rotate, translate, and scale coordinate systems
need to project world points to the image plane
all the above are modeled using 4x4 matrices and 1x4 points in homogeneous coordinates
MSU CSE 803 Fall 2008 Stockman

14. Translation of 3D point P
Point in frame 1
Point in frame 2
Point in 3D
parameters
MSU CSE 803 Fall 2008 Stockman

15. Scaling 3D point P
MSU CSE 803 Fall 2008 Stockman

16. Rotation of P about the X-axis
MSU CSE 803 Fall 2008 Stockman

17. Rotate P about the Y-axis
MSU CSE 803 Fall 2008 Stockman

18. Rotate P about the Z-axis
Looks same as 2D rotation omitting row, col 4
MSU CSE 803 Fall 2008 Stockman

19. Arbitrary rotation has orthonormal rows and columns
MSU CSE 803 Fall 2008 Stockman

20. Example: camera relative to world
MSU CSE 803 Fall 2008 Stockman

21. exercise
verify that the 3 x 3 rotation matrix is orthonormal by checking 6 dot products
invert the 3 x 3 rotation matrix
invert the 4 x 4 matrix
verify that the new matrix transforms points correctly from C to W
MSU CSE 803 Fall 2008 Stockman

22. Transformation “calculus”: notation accounts for transforms
Destination frame W
Denotes transformation W T M
T transforms points from model frame to world frame. (Notation from John Craig, 1986)
Origin frame M
MSU CSE 803 Fall 2008 Stockman

23. Apply transformations to points
Point in model coordinates W W M
P = T P M
Point in world coordinates
Transformation from model to world coordinates (instance transformation)
MSU CSE 803 Fall 2008 Stockman

24. Matrix algebra enables composition
Let M and N be 4 x 4 matrices and let P be a 4 x 1 point
M ( N P ) = ( M N ) P
we can transform P using N and then transform that by M, or we can multiply matrices M and N and then apply that to point P
matrix multiplication is associative (but not commutative)
MSU CSE 803 Fall 2008 Stockman

25. Composing transformations
Projection parms.
Parameters: rotation and translation
cancel A A C
T (w)
T =
T(p) W W C
Two transformations are composed to get one transformation that maps points from the world frame to the frame A
MSU CSE 803 Fall 2008 Stockman

26. Deriving form of the camera matrix
We have already described what the camera matrix does and what form it has; we now go through the steps to justify it
MSU CSE 803 Fall 2008 Stockman

27. Viewing model points M What’s in front of the camera?
MSU CSE 803 Fall 2008 Stockman

28. Math for the steps
Camera C maps 3D points in world W to 2D pixels in image I
MSU CSE 803 Fall 2008 Stockman

29. Perspective transformation: camera origin at the center of projection
This transformation uses same units in 3D as in the image plane
MSU CSE 803 Fall 2008 Stockman

30. Perspective projection: camera origin in the real image plane
MSU CSE 803 Fall 2008 Stockman

31. Rigid transformation for change of coordinate frame
3D coordinate frame of camera
MSU CSE 803 Fall 2008 Stockman

32. Relate camera frame to world frame
MSU CSE 803 Fall 2008 Stockman

33. Change scene units to pixels
To get into XV or GIMP image coordinates! This is a 2D to 2D transformation.
MSU CSE 803 Fall 2008 Stockman

34. Final result a 3x4 camera matrix maps 2D image pixels to 3D rays
maps 3D rays to 2D image pixels
obtain matrix via calibration (easy)
obtain matrix via reasoning (hard)
do camera calibration exercise
MSU CSE 803 Fall 2008 Stockman

35. The camera model, or matrix, is 3 x 4 and maps a homogeneous point in the world to a homogeneous pixel in the image. The ‘1’ is used to model translation
MSU CSE 803 Fall 2008 Stockman