3D Sensing Camera Model Camera Calibration

Slides:



Advertisements
Similar presentations
Epipolar Geometry.
Advertisements

Computer vision: models, learning and inference
Two-view geometry.
Computer vision. Camera Calibration Camera Calibration ToolBox – Intrinsic parameters Focal length: The focal length in pixels is stored in the.
Camera calibration and epipolar geometry
Structure from motion.
Computer Vision : CISC 4/689
MSU CSE 240 Fall 2003 Stockman CV: 3D to 2D mathematics Perspective transformation; camera calibration; stereo computation; and more.
1 3D Sensing Camera Model and 3D Transformations Camera Calibration (Tsai’s Method) Depth from General Stereo (overview) Pose Estimation from 2D Images.
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.
CS485/685 Computer Vision Prof. George Bebis
1 Introduction to 3D Imaging: Perceiving 3D from 2D Images How can we derive 3D information from one or more 2D images? There have been 2 approaches: 1.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Single-view geometry Odilon Redon, Cyclops, 1914.
3D Computer Vision and Video Computing 3D Vision Lecture 14 Stereo Vision (I) CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A
The Pinhole Camera Model
Projected image of a cube. Classical Calibration.
CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.
May 2004Stereo1 Introduction to Computer Vision CS / ECE 181B Tuesday, May 11, 2004  Multiple view geometry and stereo  Handout #6 available (check with.
CSE473/573 – Stereo Correspondence
CS 558 C OMPUTER V ISION Lecture IX: Dimensionality Reduction.
1 Perceiving 3D from 2D Images How can we derive 3D information from one or more 2D images? There have been 2 approaches: 1. intrinsic images: a 2D representation.
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
CSE 6367 Computer Vision Stereo Reconstruction Camera Coordinate Transformations “Everything should be made as simple as possible, but not simpler.” Albert.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013.
Automatic Camera Calibration
Lecture 11 Stereo Reconstruction I Lecture 11 Stereo Reconstruction I Mata kuliah: T Computer Vision Tahun: 2010.
Structure from images. Calibration Review: Pinhole Camera.
Multi-view geometry.
Lecture 12 Stereo Reconstruction II Lecture 12 Stereo Reconstruction II Mata kuliah: T Computer Vision Tahun: 2010.
Epipolar geometry The fundamental matrix and the tensor
1 Preview At least two views are required to access the depth of a scene point and in turn to reconstruct scene structure Multiple views can be obtained.
© 2005 Yusuf Akgul Gebze Institute of Technology Department of Computer Engineering Computer Vision Geometric Camera Calibration.
Course 12 Calibration. 1.Introduction In theoretic discussions, we have assumed: Camera is located at the origin of coordinate system of scene.
Geometric Models & Camera Calibration
Shape from Stereo  Disparity between two images  Photogrammetry  Finding Corresponding Points Correlation based methods Feature based methods.
Lecture 04 22/11/2011 Shai Avidan הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
3D Sensing and Reconstruction Readings: Ch 12: , Ch 13: , Perspective Geometry Camera Model Stereo Triangulation 3D Reconstruction by.
Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow.
Lecture 03 15/11/2011 Shai Avidan הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
Computer Vision Stereo Vision. Bahadir K. Gunturk2 Pinhole Camera.
Affine Structure from Motion
Single-view geometry Odilon Redon, Cyclops, 1914.
CSE 185 Introduction to Computer Vision Stereo. Taken at the same time or sequential in time stereo vision structure from motion optical flow Multiple.
Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform  Location of the impulse.
Two-view geometry. Epipolar Plane – plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections of the.
EECS 274 Computer Vision Affine Structure from Motion.
stereo Outline : Remind class of 3d geometry Introduction
1 Chapter 2: Geometric Camera Models Objective: Formulate the geometrical relationships between image and scene measurements Scene: a 3-D function, g(x,y,z)
Computer vision: models, learning and inference M Ahad Multiple Cameras
3D Reconstruction Using Image Sequence
3D Sensing 3D Shape from X Perspective Geometry Camera Model Camera Calibration General Stereo Triangulation 3D Reconstruction.
Single-view geometry Odilon Redon, Cyclops, 1914.
Digital Image Processing Additional Material : Imaging Geometry 11 September 2006 Digital Image Processing Additional Material : Imaging Geometry 11 September.
Multiple View Geometry and Stereo. Overview Single camera geometry – Recap of Homogenous coordinates – Perspective projection model – Camera calibration.
Lec 26: Fundamental Matrix CS4670 / 5670: Computer Vision Kavita Bala.
Correspondence and Stereopsis. Introduction Disparity – Informally: difference between two pictures – Allows us to gain a strong sense of depth Stereopsis.
Computer vision: geometric models Md. Atiqur Rahman Ahad Based on: Computer vision: models, learning and inference. ©2011 Simon J.D. Prince.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
Calibrating a single camera
Depth from disparity (x´,y´)=(x+D(x,y), y)
Epipolar geometry.
3D Sensing Camera Model and 3D Transformations
Course 6 Stereo.
Single-view geometry Odilon Redon, Cyclops, 1914.
3D Sensing and Reconstruction Readings: Ch 12: , Ch 13: 13
Presentation transcript:

3D Sensing Camera Model Camera Calibration 3D Shape from X Perspective Geometry Camera Model Camera Calibration General Stereo Triangulation 3D Reconstruction

3D Shape from X shading silhouette mainly research texture stereo light striping motion mainly research used in practice

Perspective Imaging Model: 1D real image point This is the axis of the real image plane. E xi f camera lens O is the center of projection. O image of point B in front image D This is the axis of the front image plane, which we use. xf zc xi xc f zc = xc B 3D object point

Perspective in 2D (Simplified) Yc camera P´=(xi,yi,f) Xc 3D object point yi P=(xc,yc,zc) =(xw,yw,zw) ray f F xi yc optical axis zw=zc xi xc f zc = xi = (f/zc)xc yi = (f/zc)yc Zc xc Here camera coordinates equal world coordinates. yi yc f zc =

3D from Stereo 3D point left image right image disparity: the difference in image location of the same 3D point when projected under perspective to two different cameras. d = xleft - xright

Depth Perception from Stereo Simple Model: Parallel Optic Axes image plane z Z f camera L xl b baseline f camera R xr x-b X P=(x,z) y-axis is perpendicular to the page. z x f xl z x-b f xr z y y f yl yr = = = =

Resultant Depth Calculation For stereo cameras with parallel optical axes, focal length f, baseline b, corresponding image points (xl,yl) and (xr,yr) with disparity d: z = f*b / (xl - xr) = f*b/d x = xl*z/f or b + xr*z/f y = yl*z/f or yr*z/f This method of determining depth from disparity is called triangulation.

Finding Correspondences If the correspondence is correct, triangulation works VERY well. But correspondence finding is not perfectly solved. (What methods have we studied?) For some very specific applications, it can be solved for those specific kind of images, e.g. windshield of a car. ° °

3 Main Matching Methods 1. Cross correlation using small windows. 2. Symbolic feature matching, usually using segments/corners. 3. Use the newer interest operators, ie. SIFT. dense sparse sparse

Epipolar Geometry Constraint: 1. Normal Pair of Images The epipolar plane cuts through the image plane(s) forming 2 epipolar lines. P z1 y1 z2 y2 epipolar plane P1 P2 x C1 b C2 The match for P1 (or P2) in the other image, must lie on the same epipolar line.

Epipolar Geometry: General Case x2 P1 e2 e1 x1 C1 C2

Constraints P Q C1 C2 1. Epipolar Constraint: Matching points lie on corresponding epipolar lines. 2. Ordering Constraint: Usually in the same order across the lines. P Q e2 e1 C1 C2

Structured Light light stripe 3D data can also be derived using a single camera a light source that can produce stripe(s) on the 3D object light source camera

Structured Light 3D Computation 3D data can also be derived using a single camera a light source that can produce stripe(s) on the 3D object 3D point (x, y, z)  (0,0,0) light source b x axis f b [x y z] = --------------- [x´ y´ f] f cot  - x´ (x´,y´,f) image 3D

Depth from Multiple Light Stripes What are these objects?

Our (former) System 4-camera light-striping stereo cameras projector rotation table 3D object

Camera Model: Recall there are 5 Different Frames of Reference yc xc Object World Camera Real Image Pixel Image zw C yf xf image a W zc yw pyramid object zp A yp xp xw

The Camera Model How do we get an image point IP from a world point P? Px Py Pz 1 s IPr s IPc s c11 c12 c13 c14 c21 c22 c23 c24 c31 c32 c33 1 = image point camera matrix C world point What’s in C?

The camera model handles the rigid body transformation from world coordinates to camera coordinates plus the perspective transformation to image coordinates. 1. CP = T R WP 2. FP = (f) CP Why is there not a scale factor here? CPx CPy CPz 1 s FPx s FPy s FPz s 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1/f 0 = 3D point in camera coordinates image point perspective transformation

Camera Calibration In order work in 3D, we need to know the parameters of the particular camera setup. Solving for the camera parameters is called calibration. yc intrinsic parameters are of the camera device extrinsic parameters are where the camera sits in the world yw xc C zc W xw zw

Intrinsic Parameters principal point (u0,v0) scale factors (dx,dy) C aspect ratio distortion factor  focal length f lens distortion factor  (models radial lens distortion) C f (u0,v0)

Extrinsic Parameters translation parameters t = [tx ty tz] rotation matrix r11 r12 r13 0 r21 r22 r23 0 r31 r32 r33 0 0 0 0 1 R = Are there really nine parameters?

Calibration Object The idea is to snap images at different depths and get a lot of 2D-3D point correspondences.

The Tsai Procedure The Tsai procedure was developed by Roger Tsai at IBM Research and is most widely used. Several images are taken of the calibration object yielding point correspondences at different distances. Tsai’s algorithm requires n > 5 correspondences {(xi, yi, zi), (ui, vi)) | i = 1,…,n} between (real) image points and 3D points.

In this* version of Tsai’s algorithm, The real-valued (u,v) are computed from their pixel positions (r,c): u =  dx (c-u0) v = -dy (r - v0) where - (u0,v0) is the center of the image - dx and dy are the center-to-center (real) distances between pixels and come from the camera’s specs -  is a scale factor learned from previous trials * This version is for single-plane calibration.

Tsai’s Procedure 1. Given the n point correspondences ((xi,yi,zi), (ui,vi)) Compute matrix A with rows ai ai = (vi*xi, vi*yi, -ui*xi, -ui*vi, vi) These are known quantities which will be used to solve for intermediate values, which will then be used to solve for the parameters sought.

Intermediate Unknowns 2. The vector of unknowns is  = (1, 2, 3, 4, 5): 1=r11/ty 2=r12/ty 3=r21/ty 4=r22/ty 5=tx/ty where the r’s and t’s are unknown rotation and translation parameters. 3. Let vector b = (u1,u2,…,un) contain the u image coordinates. 4. Solve the system of linear equations A  = b for unknown parameter vector .

Use  to solve for ty, tx, and 4 rotation parameters 5. Let U = 12 + 22 + 32 + 42 . Use U to calculate ty2 .

except… r11 = 1 ty r12 = 2 ty r21 = 3 ty r22 = 4 ty Now we know 6. Try the positive square root ty = (t 2 ) and use it to compute translation and rotation parameters. 1/2 r11 = 1 ty r12 = 2 ty r21 = 3 ty r22 = 4 ty tx = 5 ty Now we know 2 translation parameters and 4 rotation parameters. except…

Determine true sign of ty and compute remaining rotation parameters. 7. Select an object point P whose image coordinates (u,v) are far from the image center. 8. Use P’s coordinates and the translation and rotation parameters so far to estimate the image point that corresponds to P. If its coordinates have the same signs as (u,v), then keep ty, else negate it. 9. Use the first 4 rotation parameters to calculate the remaining 5.

Calculating the remaining 5 rotation parameters:

Solve another linear system. 10. We have tx and ty and the 9 rotation parameters. Next step is to find tz and f. Form a matrix A´ whose rows are: ai´ = (r21*xi + r22*yi + ty, vi) and a vector b´ whose rows are: bi´ = (r31*xi + r32*yi) * vi 11. Solve A´*v = b´ for v = (f, tz).

Almost there 12. If f is negative, change signs (see text). 13. Compute the lens distortion factor  and improve the estimates for f and tz by solving a nonlinear system of equations by a nonlinear regression. 14. All parameters have been computed. Use them in 3D data acquisition systems.

We use them for general stereo. P y1 y2 P2=(r2,c2) x2 P1=(r1,c1) e2 e1 x1 B C

For a correspondence (r1,c1) in image 1 to (r2,c2) in image 2: 1. Both cameras were calibrated. Both camera matrices are then known. From the two camera equations B and C we get 4 linear equations in 3 unknowns. r1 = (b11 - b31*r1)x + (b12 - b32*r1)y + (b13-b33*r1)z c1 = (b21 - b31*c1)x + (b22 - b32*c1)y + (b23-b33*c1)z r2 = (c11 - c31*r2)x + (c12 - c32*r2)y + (c13 - c33*r2)z c2 = (c21 - c31*c2)x + (c22 - c32*c2)y + (c23 - c33*c2)z Direct solution uses 3 equations, won’t give reliable results.

Solve by computing the closest approach of the two skew rays. V = (P1 + a1*u1) – (Q1 + a2*u2) (P1 + a1*u1) – (Q1 + a2*u2) · u1 = 0 (P1 + a1*u1) – (Q1 + a2*u2) · u2 = 0 P1 u1 solve for shortest V P Q1 u2 If the rays intersected perfectly in 3D, the intersection would be P. Instead, we solve for the shortest line segment connecting the two rays and let P be its midpoint.