High Resolution Surface Reconstruction from Overlapping Multiple-Views Nader Salman nsalman@sophia.inria.fr Mariette Yvinec myvinec@sophia.inria.fr Geometrica Team-Project JGA January 29th, 2009

Motivation Urban scenes are hard to model Complex geometry and topology Severe occlusions  We target automatic surface reconstruction from stereo vision data http://cvlab.epfl.ch/~strecha/multiview/denseMVS.html

Related Work Image-based methods Model-based methods Reconstructing 3D models of objects and scenes from N-views Image-based methods Image mosaics [Szeliski et al. 97] Lightfield [Levoy et al. 96] and Lumigraph [Cohen et al. 96] Model-based methods Active sensing approach [Turk et al. 94], [Laurendeau et al. 95], [Levoy et al. 96] Passive sensing approach Silhouette based [Niem et al. 97] Structure from motion Simple planar scenes [Zisserman et al. 96, 98] Semi-automatic reconstruction [Debevec et al. 96] Automatic reconstruction [Boissonnat et al. 88], [Taylor et al. 03], [Hilton 05], [Pons et al. 07] Image based methods : no explicit 3D model of the scene, new views are synthesized by resampling directly the captured images Image mosaics : multiple overlapping images are matched and integrated into a single panoramic image Lightfield & Lumigraph : Large number of calibrated images are captured and resampled to generate new views. Model based methods : Reconstruction of 3D surface model that can subsequently be rendered to generate new views Active sensing : project structured light pattern into the scene to recover dense 3D measurements of points on the scene surface (Range images : 2D images + depth per pixel => dense data) Passive sensing : reconstruct 3D estimates of observed scene features from multiple camera views (Sparse data) Silhouette based: Use foreground/background separation to reconstruct object shape from occluded contours. Structure from motion: Reconstruct 3D estimate of the scene structure from passive video image sequences. Planar surface fitting (Simple planar scenes) : planar surfaces can be fitted to unstructured cloud of 3D points. Semi-automatic algorithm : reconstruct scene model from sparse 3D data # volumetric primitives placed according to the 3D features of the scene and constrained optimization reconstructs a best-fit model. Automatic algorithm : reconstruct automatically scene geometry from sparse 3D data set.

System Overview Image Capture 2D Constrained Triangulation 3D Triangle Soup Triangle soup Filtering Structure from Motion Input Preprocessing Reconstruction

System Overview Image Capture 2D Constrained Triangulation 3D Triangle Soup Triangle soup Filtering Structure from Motion Input Preprocessing Reconstruction 5

Image Capture and 3D recovery (1) Video sequences captured images (100-3000 images) ~10k points Matchmoving (camera calibration & point cloud generation) [MatchMover,Voodoo/VooCAT, or Boujou]

Image Capture and 3D recovery (2) High resolution localized still pictures captured images (10-50 images) Features extraction [ENPC/Certis – CSTB] ~300k points Frames of tracked video [MatchMover, Voodoo/VooCAT, or Boujou] Hardware [Gyroviz]

Image Capture and 3D recovery (2) High resolution localized still pictures captured images (10-50 images) Features extraction [ENPC/Certis – CSTB] ~300k points Frames of tracked video [MatchMover, Voodoo/VooCAT, or Boujou] Hardware [Gyroviz] 8

Image Capture and 3D recovery (2) Matchmoving [Voodoo] Output is data dependent - Movement while shooting video - Lighting conditions Sparse point clouds Outliers ~300k points ~10k points Features extraction [ENPC/Certis – CSTB] Dense point clouds (10-20x more) Requires calibrated images Many outliers Slow 9

System Overview Image Capture 2D Constrained Triangulation 3D Triangle Soup Triangle soup Filtering Structure from Motion Input Preprocessing Reconstruction 10

Input Preprocessing (1) Outliers removal wrt average K-nearest neighbors squared distance Detected outliers Detected outliers

Input Preprocessing (2) Outliers removal wrt point error (number of cameras, cameras cone angle) Detected outliers Detected outliers

Input Preprocessing (3) 3D Points merging Point cloud smoothing (jet-fitting + reprojection) [Pouget-Cazals07]

System Overview Image Capture 2D Constrained Triangulation 3D Triangle Soup Triangle soup Filtering Structure from Motion Input Preprocessing Reconstruction 14

Reconstruction Algorithm 1 Extract 3D triangles that lie on the surface of the scene. Extract a surface out of these triangles. 2

Triangle soup extraction (1) Idea: Extract 3D triangles whose corresponding 2D does not intersect the image contour N times N images Step 3 2D constrained Tessellation Backprojected into 3D space Step 4 Step 1 Filter triangle soup Contour extraction Constraints retrieval Step 2 Triangle soup

Triangle soup extraction (2) Idea: Extract 3D triangles whose corresponding 2D does not intersect the image contour Step 1: Contour extraction Grayscale Effective luminance of a pixel Filter Gaussian smoothing mask Extract contours Frei-chen convolution masks Survey www.2d3.com

Triangle soup extraction (3) Idea: Extract 3D triangles whose corresponding 2D does not intersect the image contour Step 2: Constraints retrieval Build complete graph of feature points keep edges that respect the contour The edge weight depends on number of white pixels Adapt Needleman-Wunsch [NW1970] Filter collinear and intersecting edges Give higher priority to short edges

Triangle soup extraction (4) Idea: Extract 3D triangles whose corresponding 2D does not intersect the image contour Step 3: 2D constrained tessellation and backprojection Add constraints in 2D Delaunay tessellation Backproject into 3D space using corresponding 3D points N times N images Triangle Soup

System Overview Image Capture 2D Constrained Triangulation 3D Triangle Soup Triangle soup Filtering Structure from Motion Input Preprocessing Reconstruction 20

Triangle soup filtering (1) Problem: Triangle soup contains lots of incorrect triangles Filter triangles using size heuristic Compute bounding box of triangle soup Each triangle longest edge is compared to the half diagonal of the bounding box Filter triangles using visibility constraint [Taylor03] Empty ray in space between the camera location and 3D point position.

Triangle soup filtering (1) Before size heuristic After size heuristic 22

Triangle soup filtering (2) Problem: Triangle soup contains lots of incorrect triangles Filter triangles using size heuristic Compute bounding box of triangle soup Each triangle longest edge is compared to the half diagonal of the bounding box Filter triangles using visibility constraint [Taylor03] Empty ray in space between the camera location and 3D point position. 23

Triangle soup filtering (2) Before visibility constraint After visibility constraint 24

System Overview Image Capture 2D Constrained Triangulation 3D Triangle Soup Triangle soup Filtering Structure from Motion Input Preprocessing Reconstruction 25

Reconstruction (1) Restricted Delaunay How do we extract a surface from this polygon soup? Restricted Delaunay Compute 3D Delaunay triangulation of triangle soup vertices Compute restricted Delaunay to the union of triangles

Restricted Delaunay reconstruction Show results

Reconstruction (2) OR Restricted Delaunay Extensions: How do we extract a surface from this polygon soup? Restricted Delaunay  Triangles respect empty circle condition Non watertight surface (holes) Extensions: Remesh using restricted Delaunay OR Scatter a large number of points continuously across the surface of each triangle Create an implicit surface that approximates the triangle soup

Some results LA_Aerial www.realviz.com ~8k points 29

Some results Lion www.realviz.com ~20k points 30

Results Ours [Furokawa et al. 2007] [Strecha et al. 2006] [Pons et al. 2009] Ours 31

Future work More data acquisition and testing for automatic parameter Improve and complete existing algorithms Triangles soup algorithm Step 1 : Contour extraction Segmentation techniques [Lévy06] Histogram for automatic threshold [Asano08] Step 4 : Triangle soup filtering Some “big triangles” are correct Use circumscribing circle to differentiate between detected triangles Project triangles on respective images Use cross correlation to detect incorrect triangles Sharp features detection and reconstruction

Thank you Nader Salman http://www.gyroviz.org