Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach Hendrik Kück, Wolfgang Heidrich, Christian Vogelgsang

The goal

Our approach Perform reconstruction using Color Object’s silhouettes 1.Create initial approximation based on silhouettes (Visual Hull) 2.Improve mesh using color information within an optimization approach Perform reconstruction using Color Object’s silhouettes 1.Create initial approximation based on silhouettes (Visual Hull) 2.Improve mesh using color information within an optimization approach

Talk outline Image data & preprocessing Visual Hull Definition Computation as a triangle mesh Image based mesh optimization Results Image data & preprocessing Visual Hull Definition Computation as a triangle mesh Image based mesh optimization Results

Image data & preprocessing Calibrated images Foreground/ background segmentation Calibrated images Foreground/ background segmentation

Shape from Silhouettes Silhouette + camera information: Silhouette Cone Completely contains real object

Shape from Silhouettes Silhouette + camera information: Silhouette Cone Completely contains real object Silhouettes can have holes

Visual Hull Definition: Largest volume that produces the same silhouettes as the object Construction: Intersection of the silhouette cones Definition: Largest volume that produces the same silhouettes as the object Construction: Intersection of the silhouette cones

Visual Hull

Computing the Visual Hull Extract Visual Hull using Extended Marching Cubes algorithm (Kobbelt, Botsch, Schwanecke, Seidel, 2001) Vertices lie exactly on isosurface Can preserve sharp discontinuities Requires signed directed distance function D(VH,x,d) Distance from x to VH surface along direction d Positive, if x outside VH, negative if inside Extract Visual Hull using Extended Marching Cubes algorithm (Kobbelt, Botsch, Schwanecke, Seidel, 2001) Vertices lie exactly on isosurface Can preserve sharp discontinuities Requires signed directed distance function D(VH,x,d) Distance from x to VH surface along direction d Positive, if x outside VH, negative if inside

Computing the Visual Hull D(VH,x,d) = D(  SC i,x,d) = max i D(SC i,x,d) D(SC i,x,d) can be efficiently computed in image space D(VH,x,d) = D(  SC i,x,d) = max i D(SC i,x,d) D(SC i,x,d) can be efficiently computed in image space

Computing the Visual Hull triangles1400 triangles

The Optimization Stage Evolve triangle mesh into a shape that is Consistent with color in the images Consistent with silhouettes in images Free from self-intersections Smooth (low curvature) Composed of well shaped triangles Evolve triangle mesh into a shape that is Consistent with color in the images Consistent with silhouettes in images Free from self-intersections Smooth (low curvature) Composed of well shaped triangles

Optimization Only geometry is optimized, not topology 3 N v degrees of freedom Global optimization hopeless  Use local per-vertex optimizations Locally minimize energy function E(v i ) using 3D Simplex Method Iterate over vertices Only geometry is optimized, not topology 3 N v degrees of freedom Global optimization hopeless  Use local per-vertex optimizations Locally minimize energy function E(v i ) using 3D Simplex Method Iterate over vertices

Per Vertex Energy Function color silhouette triangle shape local curvature self penetration

Color consistency Assumption: Lambertian reflectance Surface points appear the same from all viewing directions Points on the real surface will project onto pixels of the same color in all images that see them Projecting images onto the mesh If surface is consistent, the color from different images will match Color cost term  color mismatch (L 2 norm) Assumption: Lambertian reflectance Surface points appear the same from all viewing directions Points on the real surface will project onto pixels of the same color in all images that see them Projecting images onto the mesh If surface is consistent, the color from different images will match Color cost term  color mismatch (L 2 norm)

Color consistency Use OpenGL for color projection Projective texture mapping And for determining visibility Shadow mapping v v vv

Color consistency Set up orthographic view of triangle fan around vertex v Choose scale according to sampling rate in the images Render fan to get samples of color and occlusion, once for each (relevant) image

Silhouette consistency No part of the geometry may project outside any silhouette (must stay inside the visual hull)  Strongly penalize distance outside Visual Hull ¼ max i ( distance outside silhouettes in image i ) Geometry may be smaller than Visual Hull Where color does not provide enough information, use Visual Hull as fallback solution  Slightly penalize distance inside Visual Hull ¼ min i ( distance inside silhouettes in image i ) No part of the geometry may project outside any silhouette (must stay inside the visual hull)  Strongly penalize distance outside Visual Hull ¼ max i ( distance outside silhouettes in image i ) Geometry may be smaller than Visual Hull Where color does not provide enough information, use Visual Hull as fallback solution  Slightly penalize distance inside Visual Hull ¼ min i ( distance inside silhouettes in image i )

Silhouette consistency Encode distance from silhouette in alpha channel of OpenGL textures Project onto triangle fan along with color & visibility Encode distance from silhouette in alpha channel of OpenGL textures Project onto triangle fan along with color & visibility

Multi-Resolution Optimization Local minima are a problem, especially when Triangle size is small compared to geometric error Texture frequencies are high compared to geometric error Solution: Perform optimization at multiple resolutions Local minima are a problem, especially when Triangle size is small compared to geometric error Texture frequencies are high compared to geometric error Solution: Perform optimization at multiple resolutions

Multi-Resolution Optimization Start with low resolution Visual Hull mesh

Multi-Resolution Optimization Start with low resolution Visual Hull mesh Optimize until convergence

Multi-Resolution Optimization Start with low resolution Visual Hull mesh Optimize until convergence Subdivide & optimize more

Multi-Resolution Optimization Start with low resolution Visual Hull mesh Optimize until convergence Subdivide & optimize more Do it again, …

Final Results before

Final Results beforeafter

Final Results beforeafter

Thank you

Visual Hull Properties Completely contains the object Can capture concavities and holes Can not capture indentations objectvisual hull convex hull

Related work Esteban, Schmitt (2002) Multi-stereo 3D object reconstruction Esteban, Schmitt (2003) Silhouette and stereo fusion for 3d object modeling Isidoro, Sclaroff (2003) Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints Matsuyama,Wu,Takai,Nobuhara (2003) Real-time generation and high fidelity visualization of 3D video Neumann, Aloimonos (2002) Spatio-temporal stereo using multi-resolution subdivision surfaces Esteban, Schmitt (2002) Multi-stereo 3D object reconstruction Esteban, Schmitt (2003) Silhouette and stereo fusion for 3d object modeling Isidoro, Sclaroff (2003) Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints Matsuyama,Wu,Takai,Nobuhara (2003) Real-time generation and high fidelity visualization of 3D video Neumann, Aloimonos (2002) Spatio-temporal stereo using multi-resolution subdivision surfaces

Computing the Visual Hull (1) triangles

Computing the Visual Hull (2) triangles1400 triangles

Curvature measure Sum of dihedral angles across edges

Triangle Shape Measure For one triangle 0 for equilateral triangle Approaches  for degenerate triangles E d (v) per vertex Average  over all adjacent triangles For one triangle 0 for equilateral triangle Approaches  for degenerate triangles E d (v) per vertex Average  over all adjacent triangles

Local Optimization Vertex position is optimized to minimize E(v) 3 degrees of freedom Most promising direction for optimization: Along normal vector estimate at v Corresponds to shrinking/growing the mesh Vertex position is optimized to minimize E(v) 3 degrees of freedom Most promising direction for optimization: Along normal vector estimate at v Corresponds to shrinking/growing the mesh

Local Optimization 1D Brent’s Method 3D Downhill Simplex Method Choose method depending on Local triangle shape (threshold) Success / failure of previous 1D optimization attempt 1D Brent’s Method 3D Downhill Simplex Method Choose method depending on Local triangle shape (threshold) Success / failure of previous 1D optimization attempt

Iteration Scheduling Moving a vertex affects energy of other vertices Predominantly local interactions After moving a vertex a significant distance, optimize neighbors too But also some long distance interactions All vertices have to be visited (again) Using priority queue Increase priority of neighbors after moving a vertex (  moved distance & improvement) Increase priority over time Moving a vertex affects energy of other vertices Predominantly local interactions After moving a vertex a significant distance, optimize neighbors too But also some long distance interactions All vertices have to be visited (again) Using priority queue Increase priority of neighbors after moving a vertex (  moved distance & improvement) Increase priority over time