1. Shape from Contours and Multiple Stereo A Hierarchical, Mesh-Based Approach Hendrik Kück, Wolfgang Heidrich, Christian Vogelgsang

2. The goal

4. Our approach Perform reconstruction using Color Object’s silhouettes 1.Create initial approximation based on silhouettes (Visual Hull) 2.Improve mesh using color information within an optimization approach Perform reconstruction using Color Object’s silhouettes 1.Create initial approximation based on silhouettes (Visual Hull) 2.Improve mesh using color information within an optimization approach

5. Talk outline Image data & preprocessing Visual Hull Definition Computation as a triangle mesh Image based mesh optimization Results Image data & preprocessing Visual Hull Definition Computation as a triangle mesh Image based mesh optimization Results

6. Image data & preprocessing Calibrated images Foreground/ background segmentation Calibrated images Foreground/ background segmentation

7. Shape from Silhouettes Silhouette + camera information: Silhouette Cone Completely contains real object

8. Shape from Silhouettes Silhouette + camera information: Silhouette Cone Completely contains real object Silhouettes can have holes

9. Visual Hull Definition: Largest volume that produces the same silhouettes as the object Construction: Intersection of the silhouette cones Definition: Largest volume that produces the same silhouettes as the object Construction: Intersection of the silhouette cones

10. Visual Hull

11. Computing the Visual Hull Extract Visual Hull using Extended Marching Cubes algorithm (Kobbelt, Botsch, Schwanecke, Seidel, 2001) Vertices lie exactly on isosurface Can preserve sharp discontinuities Requires signed directed distance function D(VH,x,d) Distance from x to VH surface along direction d Positive, if x outside VH, negative if inside Extract Visual Hull using Extended Marching Cubes algorithm (Kobbelt, Botsch, Schwanecke, Seidel, 2001) Vertices lie exactly on isosurface Can preserve sharp discontinuities Requires signed directed distance function D(VH,x,d) Distance from x to VH surface along direction d Positive, if x outside VH, negative if inside

12. Computing the Visual Hull D(VH,x,d) = D(  SC i,x,d) = max i D(SC i,x,d) D(SC i,x,d) can be efficiently computed in image space D(VH,x,d) = D(  SC i,x,d) = max i D(SC i,x,d) D(SC i,x,d) can be efficiently computed in image space

13. Computing the Visual Hull 14000 triangles1400 triangles

14. The Optimization Stage Evolve triangle mesh into a shape that is Consistent with color in the images Consistent with silhouettes in images Free from self-intersections Smooth (low curvature) Composed of well shaped triangles Evolve triangle mesh into a shape that is Consistent with color in the images Consistent with silhouettes in images Free from self-intersections Smooth (low curvature) Composed of well shaped triangles

15. Optimization Only geometry is optimized, not topology 3 N v degrees of freedom Global optimization hopeless  Use local per-vertex optimizations Locally minimize energy function E(v i ) using 3D Simplex Method Iterate over vertices Only geometry is optimized, not topology 3 N v degrees of freedom Global optimization hopeless  Use local per-vertex optimizations Locally minimize energy function E(v i ) using 3D Simplex Method Iterate over vertices

16. Per Vertex Energy Function color silhouette triangle shape local curvature self penetration

17. Color consistency Assumption: Lambertian reflectance Surface points appear the same from all viewing directions Points on the real surface will project onto pixels of the same color in all images that see them Projecting images onto the mesh If surface is consistent, the color from different images will match Color cost term  color mismatch (L 2 norm) Assumption: Lambertian reflectance Surface points appear the same from all viewing directions Points on the real surface will project onto pixels of the same color in all images that see them Projecting images onto the mesh If surface is consistent, the color from different images will match Color cost term  color mismatch (L 2 norm)

18. Color consistency Use OpenGL for color projection Projective texture mapping And for determining visibility Shadow mapping v v vv

19. Color consistency Set up orthographic view of triangle fan around vertex v Choose scale according to sampling rate in the images Render fan to get samples of color and occlusion, once for each (relevant) image

20. Silhouette consistency No part of the geometry may project outside any silhouette (must stay inside the visual hull)  Strongly penalize distance outside Visual Hull ¼ max i ( distance outside silhouettes in image i ) Geometry may be smaller than Visual Hull Where color does not provide enough information, use Visual Hull as fallback solution  Slightly penalize distance inside Visual Hull ¼ min i ( distance inside silhouettes in image i ) No part of the geometry may project outside any silhouette (must stay inside the visual hull)  Strongly penalize distance outside Visual Hull ¼ max i ( distance outside silhouettes in image i ) Geometry may be smaller than Visual Hull Where color does not provide enough information, use Visual Hull as fallback solution  Slightly penalize distance inside Visual Hull ¼ min i ( distance inside silhouettes in image i )

21. Silhouette consistency Encode distance from silhouette in alpha channel of OpenGL textures Project onto triangle fan along with color & visibility Encode distance from silhouette in alpha channel of OpenGL textures Project onto triangle fan along with color & visibility

22. Multi-Resolution Optimization Local minima are a problem, especially when Triangle size is small compared to geometric error Texture frequencies are high compared to geometric error Solution: Perform optimization at multiple resolutions Local minima are a problem, especially when Triangle size is small compared to geometric error Texture frequencies are high compared to geometric error Solution: Perform optimization at multiple resolutions

23. Multi-Resolution Optimization Start with low resolution Visual Hull mesh

24. Multi-Resolution Optimization Start with low resolution Visual Hull mesh Optimize until convergence

25. Multi-Resolution Optimization Start with low resolution Visual Hull mesh Optimize until convergence Subdivide & optimize more

26. Multi-Resolution Optimization Start with low resolution Visual Hull mesh Optimize until convergence Subdivide & optimize more Do it again, …

27. Final Results before

28. Final Results beforeafter

29. Final Results beforeafter

30. Thank you

31. Visual Hull Properties Completely contains the object Can capture concavities and holes Can not capture indentations objectvisual hull convex hull

32. Related work Esteban, Schmitt (2002) Multi-stereo 3D object reconstruction Esteban, Schmitt (2003) Silhouette and stereo fusion for 3d object modeling Isidoro, Sclaroff (2003) Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints Matsuyama,Wu,Takai,Nobuhara (2003) Real-time generation and high fidelity visualization of 3D video Neumann, Aloimonos (2002) Spatio-temporal stereo using multi-resolution subdivision surfaces Esteban, Schmitt (2002) Multi-stereo 3D object reconstruction Esteban, Schmitt (2003) Silhouette and stereo fusion for 3d object modeling Isidoro, Sclaroff (2003) Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints Matsuyama,Wu,Takai,Nobuhara (2003) Real-time generation and high fidelity visualization of 3D video Neumann, Aloimonos (2002) Spatio-temporal stereo using multi-resolution subdivision surfaces

33. Computing the Visual Hull (1) 24.000 triangles

34. Computing the Visual Hull (2) 14000 triangles1400 triangles

35. Curvature measure Sum of dihedral angles across edges

36. Triangle Shape Measure For one triangle 0 for equilateral triangle Approaches  for degenerate triangles E d (v) per vertex Average  over all adjacent triangles For one triangle 0 for equilateral triangle Approaches  for degenerate triangles E d (v) per vertex Average  over all adjacent triangles

37. Local Optimization Vertex position is optimized to minimize E(v) 3 degrees of freedom Most promising direction for optimization: Along normal vector estimate at v Corresponds to shrinking/growing the mesh Vertex position is optimized to minimize E(v) 3 degrees of freedom Most promising direction for optimization: Along normal vector estimate at v Corresponds to shrinking/growing the mesh

38. Local Optimization 1D Brent’s Method 3D Downhill Simplex Method Choose method depending on Local triangle shape (threshold) Success / failure of previous 1D optimization attempt 1D Brent’s Method 3D Downhill Simplex Method Choose method depending on Local triangle shape (threshold) Success / failure of previous 1D optimization attempt

39. Iteration Scheduling Moving a vertex affects energy of other vertices Predominantly local interactions After moving a vertex a significant distance, optimize neighbors too But also some long distance interactions All vertices have to be visited (again) Using priority queue Increase priority of neighbors after moving a vertex (  moved distance & improvement) Increase priority over time Moving a vertex affects energy of other vertices Predominantly local interactions After moving a vertex a significant distance, optimize neighbors too But also some long distance interactions All vertices have to be visited (again) Using priority queue Increase priority of neighbors after moving a vertex (  moved distance & improvement) Increase priority over time