1. Shape and Appearance Models from Multiple Images Richard Szeliski Microsoft Research Workshop on Image-Based Modeling and Rendering StanfordUniversity, March 24, 1998.

2. IBMR, March 24, 1998Richard Szeliski, Microsoft Research2 Image-Based Modeling u Create 3D models from one or more images –calibration and camera pose recovery –correspondence (matching, tracking, stereo) –3D model construction –appearance extraction u Mix with graphics & re-render (new views)

3. IBMR, March 24, 1998Richard Szeliski, Microsoft Research3 Image-Based Rendering u Render 3D graphics from images –sprite-based 3D rendering (Talisman) –view interpolation: warp and blend (morph) between several images –Lumigraph: full 2D manifold of images –layered depth images (LDIs): voxel-based representation

4. IBMR, March 24, 1998Richard Szeliski, Microsoft Research4 Applications u Desktop scanning for 3D world and object building (3D home page) u Collaborative design (3D fax) u Virtual environment construction (virtual tourism, home sales/redesign) u Video editing and special effects: uncalibrated, uncontrolled video

5. IBMR, March 24, 1998Richard Szeliski, Microsoft Research5 Shape and Appearance Representations u Depth maps u Volumetric models u Surface models u View-based representations u Scene decompositions: layers/sprites

6. IBMR, March 24, 1998Richard Szeliski, Microsoft Research6 Outline u Volumes from silhouettes u Surface meshes from matched curves u Depth maps from stereo u Range data merging and surface modeling u Appearance recovery (texture maps) u (Sub-) pixel-accurate, multi-view stereo u Discussion and summary

7. IBMR, March 24, 1998Richard Szeliski, Microsoft Research7 Volumes from Silhouettes u Start with collection of calibrated images

8. IBMR, March 24, 1998Richard Szeliski, Microsoft Research8 Volumes from Silhouettes u Convert images into binary silhouettes

9. IBMR, March 24, 1998Richard Szeliski, Microsoft Research9 Volumes from Silhouettes u Intersect generalized cones (using octree)

10. IBMR, March 24, 1998Richard Szeliski, Microsoft Research10 Volumes from Silhouettes Cup on turntable example

11. IBMR, March 24, 1998Richard Szeliski, Microsoft Research11 Volumes from Silhouettes u Advantages: –simple to implement, fairly robust –fast execution –complete (closed) surface u Limitations: –only produces line hull –limited resolution –sensitive to classification (thresholding)

12. IBMR, March 24, 1998Richard Szeliski, Microsoft Research12 3D Curves from Edges u Feature-based stereo matching

13. IBMR, March 24, 1998Richard Szeliski, Microsoft Research13 3D Curves from Edges u Extract extremal and internal edges

14. IBMR, March 24, 1998Richard Szeliski, Microsoft Research14 3D Curves from Edges u Match curves along epipolar lines viewing ray epipolar plane epipolar line

15. IBMR, March 24, 1998Richard Szeliski, Microsoft Research15 3D Curves from Edges u Reconstruct 3D curves… … is there a problem?

16. IBMR, March 24, 1998Richard Szeliski, Microsoft Research16 3D Curves from Edges u Silhouette curves dont match in 3D u Solution: fit circular arcs in epipolar plane

17. IBMR, March 24, 1998Richard Szeliski, Microsoft Research17 3D Curves from Edges Coffee jar example

18. IBMR, March 24, 1998Richard Szeliski, Microsoft Research18 3D Curves from Edges u Advantages: –correct estimates at occluding contours –good for smoothly curved objects –provides intrinsic surface estimates –works on interior surface markings u Limitations: –fails in highly textured regions –fails in textureless interior areas –incomplete surface (not closed)

19. IBMR, March 24, 1998Richard Szeliski, Microsoft Research19 Dense Stereo Matching u Compute depth map using correlation

20. IBMR, March 24, 1998Richard Szeliski, Microsoft Research20 Dense Stereo Matching u Move correlation windows along epipolar lines –projected window shape depends on surface orientation

21. IBMR, March 24, 1998Richard Szeliski, Microsoft Research21 Dense Stereo Matching u View extrapolation results input depth image novel view [Matthies,Szeliski,Kanade88]

22. IBMR, March 24, 1998Richard Szeliski, Microsoft Research22 Dense Stereo Matching u Newer view extrapolation results inputdepth imagenovel view

23. IBMR, March 24, 1998Richard Szeliski, Microsoft Research23 Dense Stereo Matching u Compute certainty map from correlations input depth map certainty map input depth map certainty map

24. IBMR, March 24, 1998Richard Szeliski, Microsoft Research24 Range Data Merging u Convert sparse depths to 3D points u Aggregate with certainty weighting [Soucy et al., Curless & Levoy, Puli, …]

25. IBMR, March 24, 1998Richard Szeliski, Microsoft Research25 Dense Stereo Matching u Advantages: –gives detailed surface estimates – multi-view aggregation improves accuracy u Limitations: –narrow baseline noisy estimates –fails in textureless areas –sparse, incomplete surface –sensitive to non-Lambertian effects

26. IBMR, March 24, 1998Richard Szeliski, Microsoft Research26 3D Surface Fitting u Convert 3D points into smooth surface –physically-based oriented particles [Szeliski, Tonnesen & Terzopoulos] –triangulation and mesh simplification [Hoppe et al.,...] –distance functions and isosurface extraction [Curless & Levoy]

27. IBMR, March 24, 1998Richard Szeliski, Microsoft Research27 Oriented Particles u Use a collection of small surface elements – local coordinates: position, normal, curvature –interaction potentials enforce smoothness –simulate motions using dynamics –local triangulation/interpolation scheme –topology changes occur automatically n e1e1e1e1 n e2e2e2e2 e2e2e2e2 e1e1e1e1 interaction

28. IBMR, March 24, 1998Richard Szeliski, Microsoft Research28 Oriented Particles u Interactive particle-based surface modeling

29. IBMR, March 24, 1998Richard Szeliski, Microsoft Research29 Oriented Particles u Advantages: –can conform to any topology –good for interactive shaping, design –intrinsic surface representation u Limitations: –hard to get dynamics right –slow convergence to energy minimum –non-local interactions sometimes undesirable

30. IBMR, March 24, 1998Richard Szeliski, Microsoft Research30 Texture Map Recovery u For each model patch: –determine visibility (item buffer) –blend together textures (weight by view)

31. IBMR, March 24, 1998Richard Szeliski, Microsoft Research31 Texture Map Recovery u 3D model building example octree 3D curves texture-mapped

32. IBMR, March 24, 1998Richard Szeliski, Microsoft Research32 Beyond Texture-Mapped Models u Capture view-dependent appearance –recovering BRDF [Sato et al., Yu & Malik] –view-dependent texture maps [Debevec et al.] –view interpolation [Chen & Williams, …, Seitz & Dyer] –lightfield and Lumigraph [Levoy & Hanrahan, Gortler et al.]

33. IBMR, March 24, 1998Richard Szeliski, Microsoft Research33 Lumigraph Example acquisition stagevolumetric model novel view

34. IBMR, March 24, 1998Richard Szeliski, Microsoft Research34 Multi-Image Scene Recovery u Problems with classical approach –narrow baseline noisy results –single depth map misses information –ignores (or improperly treats) occlusions –ignores mixed (partially transparent) pixels

35. IBMR, March 24, 1998Richard Szeliski, Microsoft Research35 Multi-Image Scene Recovery u Goals of new stereo algorithm –simultaneously recover disparities, colors, and opacities (c.f. blue screen matting) – explicitly handle occlusions – true multi-frame setting [Collins] –details in [Szeliski & Golland, ICCV98]

36. IBMR, March 24, 1998Richard Szeliski, Microsoft Research36 Plane Sweep Stereo u Sweep family of planes through volume –each plane defines an image composite homography virtual camera composite input image projective re-sampling of (X,Y,Z)

37. IBMR, March 24, 1998Richard Szeliski, Microsoft Research37 Plane Sweep Stereo u For each depth plane –compute composite (mosaic) image mean –compute error image variance –convert to confidence and aggregate spatially u Select winning depth at each pixel

38. IBMR, March 24, 1998Richard Szeliski, Microsoft Research38 Plane Sweep Stereo u Stack of acetates model (related to LDI...) –warp and composite (over) back-to-front layers (sprites) synthesized image

39. IBMR, March 24, 1998Richard Szeliski, Microsoft Research39 Plane Sweep Stereo u Compute visibility each input/layer pair u Recompute means, confidences, and opacities input image layer composite

40. IBMR, March 24, 1998Richard Szeliski, Microsoft Research40 Voxel Coloring u Generalizes plane sweep camera geometry –replace plane sweep with surface sweep [Seitz & Dyer][Kutulakos & Seitz]

41. IBMR, March 24, 1998Richard Szeliski, Microsoft Research41 Voxel Coloring u Results for dinosaur and rose

42. IBMR, March 24, 1998Richard Szeliski, Microsoft Research42 Stereo with Matting u Estimate fractional opacities for pixels –adjust layer sprites (colors and opacities) to best match input images –optimization criteria: F re-synthesis error F color and opacity smoothness F prior distribution on opacities –corresponds to MAP Bayesian estimator

43. IBMR, March 24, 1998Richard Szeliski, Microsoft Research43 Stereo with Matting u SRI Trees sequence example input images stereo layers

44. IBMR, March 24, 1998Richard Szeliski, Microsoft Research44 Stereo with Matting u Advantages: –true multi-image matching – deals with occlusions and mixed pixels u Limitations: –too many degrees of freedom (volume) –breaks up surfaces into voxels –no sub-pixel depths

45. IBMR, March 24, 1998Richard Szeliski, Microsoft Research45 Layered Stereo u Use arbitrarily oriented sprites [Baker,Szeliski,Anandan98] u Estimate 3D plane equation for each sprite layers (sprites)

46. IBMR, March 24, 1998Richard Szeliski, Microsoft Research46 Layered Stereo Demo u SpriteViewer: renders sprites with depth

47. IBMR, March 24, 1998Richard Szeliski, Microsoft Research47 Layered Stereo u Assign pixel to different layers (objects, sprites)

48. IBMR, March 24, 1998Richard Szeliski, Microsoft Research48 Layered Stereo u Track each layer from frame to frame, compute plane eqn. and composite mosaic u Re-compute pixel assignment by comparing original images to sprites

49. IBMR, March 24, 1998Richard Szeliski, Microsoft Research49 Layered Stereo u Resulting sprite collection

50. IBMR, March 24, 1998Richard Szeliski, Microsoft Research50 Layered Stereo u Estimated depth map

51. IBMR, March 24, 1998Richard Szeliski, Microsoft Research51 Layered Stereo u Re-synthesize original or novel images from collection of sprites

52. IBMR, March 24, 1998Richard Szeliski, Microsoft Research52 Layered Stereo u Per-pixel residual depth estimation –plane plus parallax [Anandan et al.] –model-based stereo [Debevec et al.] –model-based stereo [Debevec et al.] –better accuracy / fidelity –makes forward warping more difficult

53. IBMR, March 24, 1998Richard Szeliski, Microsoft Research53 Layered Stereo u Advantages: –can represent occluded regions –can represent transparent and border (mixed) pixels (sprites have alpha value per pixel) –works on texture-less interior regions u Limitations: –fails for high depth-complexity scenes –may need manual initialization / control

54. IBMR, March 24, 1998Richard Szeliski, Microsoft Research54 Image-Based Modeling & Rendering u Grand Unified Theory of Image-Based Modeling and Rendering u Design continuum 3D modelsimages

55. IBMR, March 24, 1998Richard Szeliski, Microsoft Research55 Modeling & Rendering u Silhouettes volume u Curves 3D mesh u Stereo depth map u Range data merging u 3D surface modeling u Texture recovery u Multi-view stereo u 3D texture-mapped model u View-dependent texture maps u Sprites with depth u Layered Depth Images u Colored depth maps u Lumigraph u Lightfield viewsobjects

56. IBMR, March 24, 1998Richard Szeliski, Microsoft Research56 Open Problems u Automatic scene segmentation u Complex scenes: forests… u Non-static scenes u Non-rigid motion u Moving illumination, specularities, … u … u … but potential of IBMR looks great

57. IBMR, March 24, 1998Richard Szeliski, Microsoft Research57 Acknowledgements u Colleagues –CMU: Takeo Kanade, Geoffrey Hinton, Larry Matthies –DEC CRL: Demetri Terzopoulos, David Tonnesen, Sing Bing Kang, James Coughlan, Richard Weiss –Microsoft: Michael Cohen, Steven Gortler, Radek Grzeszczuk, Polina Golland, Heung-Yeung Shum, Simon Baker, Anandan, Mei Han u Bibliography –see http://www.research.microsoft.com/research/vision/szeliski/IBMR

58. IBMR, March 24, 1998Richard Szeliski, Microsoft Research58 © Microsoft Corp., 1998