Computer Vision Range data Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(hurricane Isabel) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9Tracking (Welch)Optical flow Oct 14/16-- Oct 21/23Silhouettes/carving(Fall break) Oct 28/30-Structure from motion Nov 4/6Project updateProj. SfM Nov 11/13Camera calibrationSegmentation Nov 18/20FittingProb. segm.&fit. Nov 25/27Matching templates(Thanksgiving) Dec 2/4Matching relationsRange data Dec 9Final project Tentative class schedule

Computer Vision Final project Presentation: Tuesday 2-5pm, starts in SN011, then demos (make your own arrangements, preferably G-lab) Papers: due Friday (Saturday is ok, but not later!)

Computer Vision RANGE DATA Reading: Chapter 21. Active Range Sensors Segmentation Elements of Analytical Differential Geometry Registration and Model Acquisition Quaternions Object Recognition

Computer Vision Active Range Sensors Triangulation-based sensors Time-of-flight sensors New Technologies Courtesy of D. Huber and M. Hebert.

Computer Vision Structured light Single grid projection Binary code A desktop scanner

Computer Vision Principle deformation+connectivity of pattern 3D Shape Proesmans and Van Gool, ICPR96…

Computer Vision Acquisition setup Proesmans and Van Gool, ICPR96…

Computer Vision Calibration Proesmans and Van Gool, ICPR96…

Computer Vision Image with projected grid Proesmans and Van Gool, ICPR96…

Computer Vision Line detectors Proesmans and Van Gool, ICPR96…

Computer Vision Linking Proesmans and Van Gool, ICPR96…

Computer Vision Initial grid Proesmans and Van Gool, ICPR96…

Computer Vision Corrected grid sub-pixel refinement of grid Proesmans and Van Gool, ICPR96…

Computer Vision Depth computation Proesmans and Van Gool, ICPR96…

Computer Vision Removing lines Proesmans and Van Gool, ICPR96…

Computer Vision Texture estimation Proesmans and Van Gool, ICPR96…

Computer Vision 3D Reconstruction Proesmans and Van Gool, ICPR96…

Computer Vision Dionysos Proesmans and Van Gool, ICPR96…

Computer Vision Theatre mask Proesmans and Van Gool, ICPR96…

Computer Vision Capital Proesmans and Van Gool, ICPR96…

Computer Vision Coded planes Structured light: –Use projector as a camera –Figure out correspondences by coding light pattern –Only need to code 1D (but not parallel with epipolar lines!) B. Curless

Computer Vision A desktop scanner

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision More range sensors DeltaSphere Z-cam

Computer Vision Real-time system Koninckx and Van Gool

Computer Vision Elements of Analytical Differential Geometry RE Parametric surface: x : U  R 2  E 3 Unit surface normal: N = (x u x x v ) | x u x x v | 1 First fundamental form: I( t, t ) = Eu’ 2 + 2Fu’v’+Gv’ 2 E=x u. x u F=x u. x v G=x v. x v { Second fundamental form: II( t, t ) = eu’ 2 + 2fu’v’+gv’ 2 e= – N. x uu f = – N. x uv g= – N. x vv { Normal and Gaussian curvatures:  t = I( t, t ) II( t, t ) K = eg – f 2 EG – F 2

Computer Vision (1+h u 2 +h v 2 ) 2 h uu h vv –h uv 2 And the Gaussian curvature is: K =. Example: Monge Patches h u v x ( u, v ) = (u, v, h( u, v )) In this case N= ( –h u, –h v, 1) T E = 1+h u 2 ; F = h u h v,; G = 1+h v 2 e = ; f = ; g = (1+h u 2 +h v 2 ) 1/2 1 –h uu (1+h u 2 +h v 2 ) 1/2 –h uv (1+h u 2 +h v 2 ) 1/2 –h vv

Computer Vision u v N Example: Local Surface Parameterization u,v axes = principal directions h axis = surface normal In this case: h(0,0)=h u (0,0)=h v (0,0)=0 N=(0,0,1) T h uv (0,0)=0,  1 = – h uu (0,0),  2 = – h vv (0,0) h(u,v) ¼ – ½ (  1 u 2 +  2 v 2 ) Taylor expansion of order 2

Computer Vision Finding Step and Roof Edges in Range Images

Computer Vision Step Model k 1 x+h when x<0, k 2 x+c+h when x>0. { z= And, since z  ’’=0 in x  :

Computer Vision k 1 x+h when x<0, k 2 x+h when x>0. { z= Roof Model And   has a maximum value inversely proportional to  in a point x  located at a distance proportional to  from the origin.

Computer Vision Computing the Principal Directions and Curvatures Adaptive smoothing Finite-difference masks Reprinted from “Describing Surfaces,” by J.M. Brady, J. Ponce, A. Yuille and H. Asada, Proc. International Symposium on Robotics Research, H. Hanafusa and H. Inoue (eds.), MIT Press (1985).  1985 MIT.

Computer Vision Scale-Space Matching Reprinted from “Toward a Surface Primal Sketch,” By J. Ponce and J.M. Brady, in Three-Dimensional Machine Vision, T. Kanade (ed.), Kluwer Academic Publishers (1987).  1987 Kluwer Academic Publishers.

Computer Vision Segmentation into Planes via Region Growing (Faugeras & Hebert, 1986) Idea: Iteratively merge the pair of planar regions minimizing the average distance to the plane best fitting them. Reprinted from “The Representation, Recognition and Locating of 3D Objects,” by O.D. Faugeras and M. Hebert, the International Journal of Robotics Research, 5(3):27-52 (1986).  1986 Sage Publications. Reprinted by permission of Sage Publications.

Computer Vision Quaternions q = a +  q is a quaternion, R a 2 R is its real part, and R  2 R 3 is its imaginary part. Sum of quaternions: ( a+  ) + ( b+  ) ´ ( a+b ) + (  +  ) Multiplication by a scalar: ( a+  ) ´ ( a+ ) Quaternion product: ( a+  ) ( b+  ) ´ ( a b –  ¢  ) + ( a  + b  +  £  ) Conjugate: q = a +  ) q ´ a –  Operations on quaternions: Norm: | q | 2 ´ q q = q q = a 2 + |  | 2 Note: | qq’|= |q| |q’|

Computer Vision Quaternions and Rotations Let R denote the rotation of angle  about the unit vector u. Define q = cos  /2 + sin  /2 u. Then for any vector , R  = q  q. Reciprocally, if q = a + ( b, c, d ) T is a unit quaternion, the corresponding rotation matrix is:

Computer Vision The Iterative Closest Point Registration Algorithm (Besl and McKay, 1992) Key points: finding the closest-point pairs (k-d trees, caching); estimating the rigid transformation (quaternions).

Computer Vision Using Quaternions to Estimate a Rigid Transformation Problem: Find the rotation matrix R and the vector t that minimize i=1 E =  | x i ’ – R x i – t | 2. n At a minimum: 0 =  E/  t = –2  ( x i ’ – R x i – t ). n i=1 Or.. t = x’ – R x. If y i = x i –x and y i ’ = x i ’ –x’, the error is (at a minimum): i=1 E =  | y i ’ – R y i | 2 n i=1 =  | y i ’ q – q y i | 2 n i=1 =  | y i ’ – q y i q| 2 |q| 2 n Or.. E =  A i T A i with A i = 0 y i T -y i ’ T y i ’-y i [y i +y i ’] £ Linear least squares !!

Computer Vision ICP Registration Results Reprinted from “A Method for Registration of 3D Shapes,” by P.J. Besl and N.D. McKay, IEEE Trans. on Pattern Analysis and Machine Intelligence, 14(2): (1992).  1992 IEEE.

Computer Vision Initial alignment? Mostly open problem A possible approach using bitangents (Vanden Wyngaerd and Van Gool)

Computer Vision

Computer Vision

Computer Vision

Computer Vision Fusing Range Images (Curless & Levoy, 1996) Idea: Construct watertight surfaces as level sets of appropriate volumetric density functions. Reprinted from “A Volumetric Method for Building Complex Models from Range Images,” by B. Curless and M. Levoy, Proc. SIGGRAPH (1996).  1996 ACM, Inc. Included here by permission. Courtesy of M. Levoy.

Computer Vision Fusing Range Images (Curless & Levoy, 1996) Idea: Construct watertight surfaces as level sets of appropriate volumetric density functions. Reprinted from “A Volumetric Method for Building Complex Models from Range Images,” by B. Curless and M. Levoy, Proc. SIGGRAPH (1996).  1996 ACM, Inc. Included here by permission. Courtesy of M. Levoy.

Computer Vision Volumetric integration (Curless and Levoy, Siggraph´96) sensor range surfaces volume distance depth weight (~accuracy) signed distance to surface surface1 surface2 combined estimate use voxel space new surface as zero-crossing (find using marching cubes) least-squares estimate (zero derivative=minimum)

Computer Vision Fusing Range Images (Curless & Levoy, 1996) Idea: Construct watertight surfaces as level sets of appropriate volumetric density functions. Reprinted from “A Volumetric Method for Building Complex Models from Range Images,” by B. Curless and M. Levoy, Proc. SIGGRAPH (1996).  1996 ACM, Inc. Included here by permission. Courtesy of M. Levoy.

Computer Vision From volume to mesh: Marching Cubes First 2D, Marching Squares “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp

Computer Vision From volume to mesh: Marching Cubes “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp

Computer Vision From volume to mesh: Marching Cubes Improvement “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp

Computer Vision The Faugeras-Hebert Plane Matching Algorithm (1986) Key points: finding initial matches (area comparisons, binning); estimating the rigid transformation (quaternions).

Computer Vision Finding all the vectors v making an angle between  -  And  +  with a vector u.

Computer Vision Using Quaternions to Estimate a Rigid Transformation Linear least squares !!  : n ¢ x – d = 0 !  ’: n’ ¢ x’ – d’ = 0 where n’ = R n and d’ = n’ ¢ t + d. i=1 Problem: Find the rotation matrix R and the vector t that minimize i=1 E r =  | n i ’ – R n i | 2 n E t =  | d i ’ – d i – n i ’ ¢ t | 2. n and

Computer Vision Recognition Results (Faugeras & Hebert, 1986) Reprinted from “The Representation, Recognition and Locating of 3D Objects,” by O.D. Faugeras and M. Hebert, the International Journal of Robotics Research, 5(3):27-52 (1986).  1986 Sage Publications. Reprinted by permission of Sage Publications.

Computer Vision Spin Images (Johnson & Hebert, 1998) S P (Q)=(|PQ £ n|, PQ ¢ n)  

Computer Vision Reprinted from “Using Spin Images for Efficient Object Recognition from Cluttered 3D Scenes,” by A.E. Johnson and M. Hebert, IEEE Trans. on Pattern Analysis and Machine Intelligence, 21(5): (1999).  1999 IEEE. Sample Spin Images Matching Criterion:

Computer Vision Recognition Results Reprinted from “Using Spin Images for Efficient Object Recognition from Cluttered 3D Scenes,” by A.E. Johnson and M. Hebert, IEEE Trans. on Pattern Analysis and Machine Intelligence, 21(5): (1999).  1999 IEEE.

Computer Vision Computer Vision What next? Related courses: –Comp 254 Image Analysis (Spring) –Comp 255 Recent Advances in Image Analysis (Odd Falls) –Comp 290 3D photography?

Computer Vision The future is bright Computation is cheap Lots of pix –cameras are cheap, many pix are digital Lots of demand for “slicing and dicing” pix –generate models –new movies from old –search Lots of “hidden value” –can’t do data mining for collections with pix in them e.g. mortgage papers, cheques, etc. e.g. filtering

Computer Vision There are lots of cameras! surveillance cameras ~1500/sq.mile in Manhattan

Computer Vision Recent flowering of vision can do (sort of!) –structure from motion –segmentation –video representation –model building –tracking –face finding will be able to do (sort of!) –face recognition –inference about people –character recognition –perhaps more

Computer Vision Big open problems Next step in structure from motion Really good missing variable formalism Decent understanding of illumination, materials and shading Segmentation Representation for recognition Efficient management of relations Recognition processes for lots of objects A lot of this looks like applied statistics

Computer Vision Next week: Final project presentations