MSU CSE 803 Fall 2008 Stockman1 CV: 3D sensing and calibration Coordinate system changes; perspective transformation; Stereo and structured light

MSU CSE 803 Fall 2008 Stockman2 roadmap using multiple cameras using structured light projector 3D transformations general perspective transformation justification of 3x4 camera model

MSU CSE 803 Fall 2008 Stockman3 Four Coordinate frames W: world, C,D: cameras, M: object model Need to relate all to each other.

MSU CSE 803 Fall 2008 Stockman4 Can we recognize? Is there some object M That can be placed in some location That will create the two images that are observed? Discover/compute what object and what pose

MSU CSE 803 Fall 2008 Stockman5 Need to relate frames to compute relate camera to world using rotations and translations project world point into real image using projection scale image point in real image plane to get pixel array coordinates

MSU CSE 803 Fall 2008 Stockman6 Stereo configuration 2 corresponding image points enable the intersection of 2 rays in W

MSU CSE 803 Fall 2008 Stockman7 Stereo computation

MSU CSE 803 Fall 2008 Stockman8 Math for stereo computations need to calibrate both cameras to W so that rays in x,y,z reference same space need to have corresponding points find point of closest approach of the two rays (rays are too far apart  point correspondence error or crude calibration)

MSU CSE 803 Fall 2008 Stockman9 Replace camera with projector Can calibrate a projector to W easily. Correspondence now means identifying marks.

MSU CSE 803 Fall 2008 Stockman10 Advantages/disadvantages of structured light

MSU CSE 803 Fall 2008 Stockman11 Grid projected on objects All grid intersects are integral

MSU CSE 803 Fall 2008 Stockman12 Computing surface normals Surface normals have been computed and then added to the image (augmented reality)

MSU CSE 803 Fall 2008 Stockman13 Relating coordinate frames need to relate camera frame to world need to rotate, translate, and scale coordinate systems need to project world points to the image plane all the above are modeled using 4x4 matrices and 1x4 points in homogeneous coordinates

MSU CSE 803 Fall 2008 Stockman14 Translation of 3D point P parameters Point in 3D Point in frame 1 Point in frame 2

MSU CSE 803 Fall 2008 Stockman15 Scaling 3D point P

MSU CSE 803 Fall 2008 Stockman16 Rotation of P about the X-axis

MSU CSE 803 Fall 2008 Stockman17 Rotate P about the Y-axis

MSU CSE 803 Fall 2008 Stockman18 Rotate P about the Z-axis Looks same as 2D rotation omitting row, col 4

MSU CSE 803 Fall 2008 Stockman19 Arbitrary rotation has orthonormal rows and columns

MSU CSE 803 Fall 2008 Stockman20 Equivalent rotation form every rotation or combination of rotations is equivalent to ONE rotation of angle theta about a single axis or rotation the axis of rotation and angle of rotation can be determined if three points are known before and after

MSU CSE 803 Fall 2008 Stockman21 Computing angle-axis form A A* O B To rotate A to A*, the axis of rotation is OB that is perpendicular to OA and OA* Compute OB = OA x OA* (unit vectors) Compute  from OA o OA* (unit vectors) See frame software 

MSU CSE 803 Fall 2008 Stockman22 Example: camera relative to world

MSU CSE 803 Fall 2008 Stockman23 exercise verify that the 3 x 3 rotation matrix is orthonormal by checking 6 dot products invert the 3 x 3 rotation matrix invert the 4 x 4 matrix verify that the new matrix transforms points correctly from C to W

MSU CSE 803 Fall 2008 Stockman24 Transformation “calculus”: notation accounts for transforms T W M Denotes transformation Origin frame M Destination frame W T transforms points from model frame to world frame. (Notation from John Craig, 1986)

MSU CSE 803 Fall 2008 Stockman25 Apply transformations to points T M W P M P = W Point in model coordinates Point in world coordinates Transformation from model to world coordinates (instance transformation)

MSU CSE 803 Fall 2008 Stockman26 Matrix algebra enables composition Let M and N be 4 x 4 matrices and let P be a 4 x 1 point M ( N P ) = ( M N ) P we can transform P using N and then transform that by M, or we can multiply matrices M and N and then apply that to point P matrix multiplication is associative (but not commutative)

MSU CSE 803 Fall 2008 Stockman27 Composing transformations T (  W C T(  C A T = W A Two transformations are composed to get one transformation that maps points from the world frame to the frame A Parameters: rotation and translation Projection parms. cancel

MSU CSE 803 Fall 2008 Stockman28 Use of transformation calculus Alignment of two congruent triangles (ABC and DEF) via coordinate transformation; Fundamental operation

MSU CSE 803 Fall 2008 Stockman29 Align corresponding vertices at origin and align first side 1) Translate vertex to origin (do both) 2) Rotate DE to X-axis and AB to X-axis 0) starting configurations in 3D

MSU CSE 803 Fall 2008 Stockman30 Alignment: last steps 3) Rotate point F about X-axis into XY-plane. Rotate point C in a similar manner. Both triangles should now overlap in 3D

MSU CSE 803 Fall 2008 Stockman31 Here’s what was done Every transformation we applied is invertible. So, we get the inverses to apply to one side. This is T M W

MSU CSE 803 Fall 2008 Stockman32 Align two congruent triangles in 3D space

MSU CSE 803 Fall 2008 Stockman33 What use is it? fundamental step in aligning a 3D model to observed 3D points might be lucky to have 3 corners of a rectangular box could have any matching 3D feature points once aligned, other 3D points could be transformed the same way See /user/stockman/Software/RST/Frames/frame.cpp and look for function align_triangles

MSU CSE 803 Fall 2008 Stockman34 Deriving form of the camera matrix We have already described what the camera matrix does and what form it has; we now go through the steps to justify it

MSU CSE 803 Fall 2008 Stockman35 Viewing model points M What’s in front of the camera?

MSU CSE 803 Fall 2008 Stockman36 Math for the steps

MSU CSE 803 Fall 2008 Stockman37 Perspective transformation: camera origin at the center of projection This transformation uses same units in 3D as in the image plane

MSU CSE 803 Fall 2008 Stockman38 Perspective projection: camera origin in the real image plane

MSU CSE 803 Fall 2008 Stockman39 Rigid transformation for change of coordinate frame

MSU CSE 803 Fall 2008 Stockman40 Relate camera frame to world frame

MSU CSE 803 Fall 2008 Stockman41 Change scene units to pixels To get into XV or GIMP image coordinates!

MSU CSE 803 Fall 2008 Stockman42 Final result a 3x4 camera matrix maps 2D image pixels to 3D rays maps 3D rays to 2D image pixels obtain matrix via calibration (easy) obtain matrix via reasoning (hard) do camera calibration exercise