Convergence of vision and graphics Jitendra Malik University of California at Berkeley Jitendra Malik University of California at Berkeley
Overview 3D capture: Modeling, simulation RenderingDisplay Applications: Simulation Virtual Reality Remote collaboration
Graphics and Vision Computer graphics is the forward problem: given scene geometry, reflectances and lighting, synthesize an image. Computer vision must address the inverse problem: given an image/multiple images, reconstruct the scene geometry, reflectacnes and illumination. Computer graphics is the forward problem: given scene geometry, reflectances and lighting, synthesize an image. Computer vision must address the inverse problem: given an image/multiple images, reconstruct the scene geometry, reflectacnes and illumination.
Image-based Modeling –Vary viewpoint –Vary lighting –Vary scene configuration –Vary viewpoint –Vary lighting –Vary scene configuration Recover Models of Real World Scenes and Make Possible Various Visual Interactions
Image-based Modeling 1st Generation---- vary viewpoint but not lighting –Acquire photographs –Recover geometry (explicit or implicit) –Texture map 1st Generation---- vary viewpoint but not lighting –Acquire photographs –Recover geometry (explicit or implicit) –Texture map
Recovering geometry Historical roots in photogrammetry and analysis of 3D cues in human vision Single images adequate given knowledge of object class Multiple images make the problem easier, but not trivial as corresponding points must be identified. Historical roots in photogrammetry and analysis of 3D cues in human vision Single images adequate given knowledge of object class Multiple images make the problem easier, but not trivial as corresponding points must be identified.
Arc de Triomphe
The Taj Mahal Taj Mahal modeled from one photograph by G. Borshukov
Campus Model of UC Berkeley Campanile + 40 Buildings (Debevec et al)
Image-based Modeling 2nd Generation---- vary viewpoint and lighting –Recover geometry & reflectance properties –Render using light transport simulation or local shading 2nd Generation---- vary viewpoint and lighting –Recover geometry & reflectance properties –Render using light transport simulation or local shading Original Lighting & ViewpointNovel Lighting & Viewpoint
Inverse Global Illumination (Yu et al) Reflectance Properties Radiance Maps Geometry Light Source s
Real vs. Synthetic
Image-based Modeling 3 rd Generation--Vary spatial configurations in addition to viewpoint and lighting Novel ViewpointNovel Viewpoint & Configuration
Our Framework Input –Multiple range scans of a scene –Multiple photographs of the same scene Input –Multiple range scans of a scene –Multiple photographs of the same scene Output –Geometric meshes of each object in the scene –Registered texture maps for objects
Segmentation: From images to objects
Segmentation Results [Yu, Ferencz and Malik ’00]
Models of Individual Objects
Texture-Mapping and Object Manipulation
Image Based modeling for motion capture Body Suits, Markers Video Motion Capture
Eadweard Muybridge [Bregler and Malik ’98]
Continuing Challenges Finding correspondences automatically Optimal estimation of structure from n views under perspective projection Models of reflectance and texture for natural materials and objects Finding correspondences automatically Optimal estimation of structure from n views under perspective projection Models of reflectance and texture for natural materials and objects