Acquiring graphical models of shape and motion James Davis ISM101 May 2005

2 Graphical models often represent some aspect of the real world Shape MotionSurface Reflectance Light

3 Graphical models are the bridge between people and computation World of People World of Computation Acquire Visualize Graphical Models

4 Acquired graphical models are important outside of computer science Movies Communication Biomechanics Medicine Art history Games ArcheologyAnthropology Graphical Models

5 Laser scanners locate a bright intensity peak in the image plane and triangulate depth xixi Laser Camera

6

7

8 How do we capture motion in large working volumes and at extreme detail?

9 Adding more and more cameras increases resolution but results in scaling difficulties

10 We can exploit the non-uniform resolution characteristics of most interesting motion

11 A low resolution sub-system can be used to direct high resolution pan-tilt cameras [Davis and Chen - ICRA Foveated observation of shape and motion – Best Vision Paper Award] [Davis and Chen - ICCV Calibrating Pan-tilt Cameras in Wide-area Surveillance Networks] [Chen and Davis - Submitted An occlusion metric for selecting robust camera configurations] [Davis – Ph.D. dissertation Mixed Scale Motion Recovery] [Chen and Davis – CVPR Wide Area Camera Calibration Using Virtual Calibration Objects]

12

13

14 Desktop laser scanners can measure the shape of small objects ~ 15 cm

15 The Digital Michelangelo project obtained very large models at very high resolution [Levoy, Pulli, Curless, Rusinkiewicz, Koller, Pereira, Ginzton, Anderson, Davis, Ginsberg, Shade, Fulk – Siggraph The Digital Michelangelo Project: 3D scanning of large statues ]

1 mm

18 These models have had a major impact both inside and outside computer graphics

19

20 Holes inevitably remain in scanned models

21 Scanned geometry often has complex holes

22 Locate hole boundaries and triangulate?

23 Triangulating boundaries sometimes fails Self intersecting surface

24 Hole boundaries must be correctly connected Fill hole on blue boundary - no solution possible Fill hole between blue and red boundary - solution possible Blue boundary Red boundary

25 Topological complexity is often present

26 Volumetric surface representation Surface is the zero set of a filtered sidedness function ( or equivalently a clamped signed-distance function )

27 The computational domain is limited to a narrow band around the surface Brown is unknown or unimportant region

28 Surface holes are unknown regions Brown is unknown or unimportant region

29 Diffuse the known information to fill in missing volumetric regions

30 Examples on 2D synthetic holes

31 Examples from real meshes

32 Flexible but not always correct topology

33 Scanner line of sight constraint scanner region known to be empty

34 Line of sight constraint enforces correct topology

35 Complex geometry, complex topology, guaranteed manifold surface, efficient, simple [Davis, Marschner, Garr and Levoy – 3DPVT Filling Holes in Complex Surfaces Using Volumetric Diffusion]

36 Real StatueOur Model Computer Graphics Our Model Physical Replica Purchased Replica

37

38 Existing shape capture technologies have a wide range of difficulties Moving objects Unexpected illumination Calibration

39 Traditional classification of range sensing methods prioritizes “active vs passive” [Slide from SIGGRAPH Course on 3D Photography – Curless 2000 ]

40 Laser scanners locate a bright intensity peak in the image plane and triangulate depth xixi Laser Camera

41 Passive stereo searches for corresponding image regions and triangulates depth epipolar line

42 Laser scanning matches a spatial window between a virtual camera and a real camera xixi Virtual camera Camera Virtual camera’s “view”Real camera’s view

43 Matching between the virtual and real camera is not robust to unexpected illumination Virtual camera’s “view”Real camera’s view

44 Replacing the virtual camera with a second real camera increases robustness Camera 1 Camera 2 Virtual Camera xixi Camera 1 Camera 2

45 Some laser scanners locate a bright intensity peak in over time and triangulate depth t1t1 t2t2 t3t3 t4t4 t1t1 t2t2 t3t3 t4t4 xixi

46 Temporal laser scanners can also be described in terms of corresponding vectors Virtual camera’s “view”Real camera’s view Time

47 Any unstructured lighting variation is sufficient to create a temporal matching vector

48 Any unstructured lighting variation is sufficient to create a temporal matching vector

49 Spacetime stereo searches for corresponding spacetime volumes in video sequences

50 Spacetime stereo searches for corresponding spacetime volumes in video sequences Time

51 Spacetime stereo searches for corresponding spacetime volumes in video sequences Time

52 It is possible to use both space and time Existing techniques use only one or the other Using two real cameras allows unstructured lighting variation to be matched Existing range scanners have required strictly controlled lighting

53 Version 1 Hand held light source Working volume ~ 500mm Version 2 Projector light Working volume ~ 440mm Version 3 Projector light Working volume ~ 2500mm

54 Dynamic results with spacetime matching window : 7 Horz x 1 Vert x 7 Time

mm Accurate results are possible – mm RMS error – 0.13 mm peak noise

56 Analysis of optimum size for spacetime matching window Error Temporal matching size 1 Horz x 1 Vert x {Long} Time 3 Horz x 3 Vert x 8 Time Object is not moving Object rotates 0.3 deg per frame

57 Moving objects Unexpected illumination Calibration [Davis, Ramamoothi and Rusinkiewicz – CVPR Spacetime Stereo : A Unifying Framework for Depth from Triangulation] [Davis and Chen – 3DIM A Laser Range Scanner Designed for Minimum Calibration Complexity]

58 Summary of topics and contributions Mixed Scale Motion Recovery Allows motion capture in much larger working volumes than were previously possible Digital Michelangelo Acquired most dense scanned models in existence Significantly impacted research both inside and outside computer graphics Hole filling Robust to geometric and topological complexity Guarantees a manifold non-self-intersecting surface Space-time stereo Allows recovery of dynamic scenes Robust to lighting interference

59 [Jointly with Drago Anguelov, Praveen Srinivasan, Hoi-Cheung Pang, Daphne Koller, and Sebastian Thrun] Coming soon : Determine human motion and skeletal topology from dynamic shape

60 Coming soon : Robotic localization and 3D map building [Jointly with James Diebel, Kjell Reuterswärd, Rakesh Gupta and Sebastian Thrun]

61 Coming soon : Geographic surveying of Half Dome at 20 cm resolution

62 Future work : The 95% of nature for which graphical models are still unmeasurable

63 Acknowledgements Gene Alexander Maneesh Agrawala Sean Anderson Drago Anguelov Chris Bregler Sergey Brin Xing Chen Erika Chuang Brian Curless James Diebel Duane Fulk Hector Garcia-Molina Matt Garr Jeremy Ginsberg NSF, DARPA, DOE, Intel, Sony, Interval, Honda Hoi-Cheung Pang Lucas Pereira Zoran Popović Kari Pulli Ravi Ramamoothi Kjell Reuterswärd Szymon Rusinkiewicz David Salesin Jonathan Shade Philipp Slusallek Praveen Srinivasan Sebastian Thrun Matt Ginzton Mike Gleicher Hector Gonzales-Banos Rakesh Gupta Pat Hanrahan Daphne Koller David Koller Venkat Krishnamurthy Clay Kunz Marc Levoy Stephen Marschner Diego Nehab Victor Ng-Thow-Hing