« Uncalibrated Vision based on Structured Light » Joaquim Salvi 2 David Fofi 1 El Mustapha Mouaddib 3 3 CREA EA 3299 Université de Picardie Jules Verne Amiens, France 2 VICOROB - IIiA Universitat de Girona Girona, España 1 Le2i UMR CNRS 5158 Université de Bourgogne Le Creusot, France

0. Outline 1.Introduction 2.Tools for uncalibrated vision 3.Uncalibrated reconstruction 4.Experimental results 5.Conclusion ……………………………

I. Introduction 1.Structured light vision 2.Calibration vs uncalibration ……………………………

IMAGE PLANE PATTERN FRAME « Structured light vision » ……………….. J. Salvi, J. Batlle, E. Mouaddib, "A robust-coded pattern projection for dynamic measurement of moving scenes", Pattern Recognition Letters, 19, pp , J. Batlle, E. Mouaddib, J. Salvi, "Recent progress in coded structured light to solve the correspondence problem. A survey", Pattern Recognition, 31(7), pp , ……………………...……………………………

DRAWBACKS OF HARD-CALIBRATION: Off-line process (calibration pattern, etc.) Has to be repeated each time one of the parameters is modified Working with a camera with automatic focus and aperture is NOT possible. Visual adaptation to the environment is not allowed! A slide or LCD projector needs to be focused. RECONSTRUCTION FROM UNCALIBRATED SENSOR... « Calibration vs uncalibration » ……………………...……………………………

II. Tools for uncalibrated vision 1.Test of spatial colinearity 2.Test of coplanarity 3.Stability of the cross-ratio 4.Validity of the affine model ……………………………

« Test of spatial colinearity » S R Q P Cross-ratio within the pattern and cross-ratio within the image are equals if the points are colinear. ……………………...……………………………

« Test of coplanarity » p o'p' q'r' s' o qr s {o;p,q,r,s}={o';p',q',r'} Cross-ratio within the pattern and cross-ratio within the image are equals if the point are colinear. ……………………...……………………………

« Stability of the cross-ratio » Error on cross-ratios with a noise from 0 to 0.5d (d is the distance between two successive points) Nota: to compare cross-ratios a projective distance is necessary. Method of the random cross-ratios. ……………….. K. Aström, L. Morin, "Random cross-ratios", Research Report n°rt 88 imag-14, LIFIA, ……………………...……………………………

« Validity of the affine model » ……………………...…………………………… affine projection m n m' n' Valid if  0

III. Uncalibrated reconstruction 1.Projective reconstruction 2.Structured light limitations 3.Euclidean constraints through structured lighting ……………………………

Recover the scene structure from n images and m points and... Intrinsic parameters Extrinsic parameters Scene geometry Points matching    PROJECTIVE RECONSTRUCTION « Projective reconstruction » ……………………...……………………………

PARAMETERS ESTIMATION APPROACH, CANONICAL REPRESENTATION « Structured light limitations » ……………….. R. Mohr, B. Boufama, P. Brand, “Accurate projective reconstruction”, Proc. of the 2 nd ESPRIT-ARPA-NSF Workshop on Invariance, Azores, pp , Q.-T. Luong, T. Viéville, "Canonical representations for the geometries of multiple projective views", Proc. of the 3 rd Euro. Conf. on Computer Vision, Stockholm (Sweden), = MOVEMENT OF THE PROJECTOR MOVEMENT OF THE 3-D POINTS THE PATTERN SLIDES ALONG THE OBJECTS RECONSTRUCTION FROM TWO VIEWS (i.e. one view and one pattern projection) CAMERA + PROJECTOR HETEROGENEITY OF THE SENSOR INTRINSIC PARAMETERS CANNOT BE ASSUMED CONSTANT ……………………...……………………………

n images composed by m points... p ij : image point A j : projection matrix P j : object point (U ij, V ij ) : pixel co-ordinates « The parameters estimation approach » ……………………...……………………………

A unique solution cannot be performed because... W is a 4x4 invertible matrix… a collineation of the 3-D space 4x4 - 1 (scale factor) = 15 degrees of freedom, thus... 5 co-ordinates object points assigned to AN ARBITRARY PROJECTIVE BASIS. A RECONSTRUCTION WITH RESPECT TO A PROJECTIVE FRAME (distances, angles, parallelism are not preserved) « The parameters estimation approach » ……………………...……………………………

Euclidean transformations form a sub-group of projective transformations... A collineation W upgrades projective reconstruction to Euclidean one. TRANSLATING EUCLIDEAN KNOWLEDGE OF THE SCENE INTO MATHEMATICAL CONSTRAINTS ON THE ENTRIES OF W. Matching projective points with their corresponding Euclidean points ? YES, BUT... Euclidean co-ordinates of points are barely available… … if they are: pattern cross-points have to be projected exactly onto these object points. « From projective to Euclidean » ……………….. B. Boufama, R. Mohr, F. Veillon, "Euclidean constraints for uncalibrated reconstruction", Proc. of the 4 th Int. Conf. on Computer Vision, Berlin (Germany), pp , ……………………...……………………………

PROJECTED SQUARE ONTO A PLANAR SURFACE IMAGE CAPTURE A B C D The sensor behaviour is assumed to be affine... « Parallelogram constraints » ……………………...……………………………

Pattern Vert. plane Horiz. plane Points belonging to horizontal or vertical plane... Arbitrary distance between two planes... Cross-point as origin… « Alignment constraints » ……………………...……………………………

otherwise… reduced orthogonality constraint: « Orthogonality constraints » ……………………...…………………………… A'B' ·A'C' = (x A' - x B' )(x A' - x C' )+ (y A' - y B' )(y A' - y C' )+ (z A' - z B' )(z A' -z C' ) = 0 (x A' - x B' )(x A' - x C' )+ (y A' - y B' )(y A' - y C' ) = 0 Light stripes Light planes Projected lines Planar surfaces A C B A' B' C'

« Example » ……………………...…………………………… An alignment constraint : x A' = x B' (relation between unknown Euclidean points) We have: [x A' ; y A' ; z A' ; t A' ] T = W· [x A ; y A ; z A ; t A ] T [x B' ; y B' ; z B' ; t B' ] T = W· [x B ; y B ; z B ; t B ] T Then: W 1i ·x A = W 1i ·x B (relation between known projective points) … same way for the other constraints… The set of equations is solved by a non-linear optimisation method as Levenberg-Marquardt. 15 independent constraints are necessary (W is a 4  4 matrix defined up to a scale factor)

IV. Experimental results 1.Colinearity 2.Coplanarity 3.Euclidean reconstruction ……………………………

« Colinearity » Theoretical (pattern) cross-ratio = Measured (image) cross-ratio = Projective error =6.9  Decision = the points are colinear Theoretical cross-ratio = Measured cross-ratio = Projective error =6.2  Decision = the points are not colinear ……………………...……………………………

« Coplanarity » Theoretical cross-ratio = 2 Measured cross-ratio = 1.96 Projective error =2.2  Decision = the points are coplanar Theoretical cross-ratio = 2 Measured cross-ratio = Projective error =5.9  Decision = the points are not coplanar ……………………...……………………………

« Euclidean reconstruction: synthetic data » re-projection of 3D points onto the image planes (circles: synthetic points, crosses: re-projections) ……………………...……………………………

« Euclidean reconstructions » ……………………...……………………………

V. Conclusion ……………………………

Projective reconstruction from a single pattern projection and a single image capture. Pattern projection used to retrieve geometrical knowledge of the scene: uncalibrated Euclidean reconstruction. Structured lighting ensures there is known scene structure. Structured light provides numerous contraints. Tests of colinearity and coplanarity can be used to retrieve projective basis (5 points, no 4 of them being coplanar, no 3 of them being colinear).