Three Dimensional Model Construction for Visualization Avideh Zakhor Video and Image Processing Lab University of California at Berkeley

Slides:

Advertisements

Similar presentations

Bayesian Belief Propagation

Advertisements

For Internal Use Only. © CT T IN EM. All rights reserved. 3D Reconstruction Using Aerial Images A Dense Structure from Motion pipeline Ramakrishna Vedantam.

Maintainable 3D Models of Cities Gerhard NAVRATIL Rizwan BULBUL Andrew U. Frank Vienna University of Technology Institute of Geoinformation and Cartography.

Dana Cobzas-PhD thesis Image-Based Models with Applications in Robot Navigation Dana Cobzas Supervisor: Hong Zhang.

Reconstructing Building Interiors from Images Yasutaka Furukawa Brian Curless Steven M. Seitz University of Washington, Seattle, USA 2011/01/16 蔡禹婷.

Constructing immersive virtual space for HAI with photos Shingo Mori Yoshimasa Ohmoto Toyoaki Nishida Graduate School of Informatics Kyoto University GrC2011.

A new approach for modeling and rendering existing architectural scenes from a sparse set of still photographs Combines both geometry-based and image.

Advanced Computer Graphics CSE 190 [Spring 2015], Lecture 14 Ravi Ramamoorthi

Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004.

Copyright  Philipp Slusallek Cs fall IBR: Model-based Methods Philipp Slusallek.

1 MURI review meeting 09/21/2004 Dynamic Scene Modeling Video and Image Processing Lab University of California, Berkeley Christian Frueh Avideh Zakhor.

Advanced Computer Graphics (Fall 2010) CS 283, Lecture 16: Image-Based Rendering and Light Fields Ravi Ramamoorthi

Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi

Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2005 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.

C C V C L Sensor and Vision Research for Potential Transportation Applications Zhigang Zhu Visual Computing Laboratory Department of Computer Science City.

Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.

Representations of Visual Appearance COMS 6160 [Spring 2007], Lecture 4 Image-Based Modeling and Rendering

Image-Based Modeling and Rendering CS 6998 Lecture 6.

Mosaic-Based 3D Scene Representation and Rendering Zhigang Zhu Visual Computing Lab Department of Computer Science City College and Graduate Center City.

Automated 3D Model Construction for Urban Environments Christian Frueh John Flynn Avideh Zakhor Next Generation 4D Distributed Modeling and Visualization.

View interpolation from a single view 1. Render object 2. Convert Z-buffer to range image 3. Re-render from new viewpoint 4. Use depths to resolve overlaps.

3D from multiple views : Rendering and Image Processing Alexei Efros …with a lot of slides stolen from Steve Seitz and Jianbo Shi.

Image Stitching and Panoramas

Copyright  Philipp Slusallek IBR: View Interpolation Philipp Slusallek.

CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai.

 Marc Levoy History of computer graphics CS Introduction to Computer Graphics Autumn quarter, 2003 Slides for September 25 lecture.

Siggraph’2000, July 27, 2000 Jin-Xiang Chai Xin Tong Shing-Chow Chan Heung-Yeung Shum Microsoft Research, China Plenoptic Sampling SIGGRAPH’2000.

Computational Photography Light Field Rendering Jinxiang Chai.

CSCE 641: Computer Graphics Image-based Rendering Jinxiang Chai.

 Marc Levoy IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry.”

Convergence of vision and graphics Jitendra Malik University of California at Berkeley Jitendra Malik University of California at Berkeley.

 Marc Levoy IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry.”

Constructing immersive virtual space for HAI with photos Shingo Mori Yoshimasa Ohmoto Toyoaki Nishida Graduate School of Informatics Kyoto University GrC2011.

View interpolation from a single view 1. Render object 2. Convert Z-buffer to range image 3. Re-render from new viewpoint 4. Use depths to resolve overlaps.

Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.

Image Based Rendering And Modeling Techniques And Their Applications Jiao-ying Shi State Key laboratory of Computer Aided Design and Graphics Zhejiang.

The Story So Far The algorithms presented so far exploit: –Sparse sets of images (some data may not be available) –User help with correspondences (time.

The University of Ontario CS 4487/9587 Algorithms for Image Analysis n Web page: Announcements, assignments, code samples/libraries,

CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai.

© 2006 Autodesk1 Image-processing Technologies for Digital Content Creation

Real-Time High Quality Rendering CSE 291 [Winter 2015], Lecture 6 Image-Based Rendering and Light Fields

Integration Of CG & Live-Action For Cinematic Visual Effects by Amarnath Director, Octopus Media School.

A Brief Overview of Computer Vision Jinxiang Chai.

Advanced Computer Graphics (Spring 2013) CS 283, Lecture 15: Image-Based Rendering and Light Fields Ravi Ramamoorthi

Peter Sturm INRIA Grenoble – Rhône-Alpes (Institut National de Recherche en Informatique et Automatique) An overview of multi-view stereo and other topics.

Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University.

KinectFusion : Real-Time Dense Surface Mapping and Tracking IEEE International Symposium on Mixed and Augmented Reality 2011 Science and Technology Proceedings.

Interactive Virtual Relighting and Remodelling of Real Scenes C. Loscos 1, MC. Frasson 1,2,G. Drettakis 1, B. Walter 1, X. Granier 1, P. Poulin 2 (1) iMAGIS*

Image Based Rendering(IBR) Jiao-ying Shi State Key laboratory of Computer Aided Design and Graphics Zhejiang University, Hangzhou, China

MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of.

Image-based rendering Michael F. Cohen Microsoft Research.

November 10, 2004 Prof. Christopher Rasmussen Lab web page: vision.cis.udel.edu.

STREET VIEW Wenyang Liu GMAT9205: Fundamentals of Geo-Positioning Session 1, 2010.

A Camera-Projector System for Real-Time 3D Video Marcelo Bernardes, Luiz Velho, Asla Sá, Paulo Carvalho IMPA - VISGRAF Laboratory Procams 2005.

1 Howard Schultz, Edward M. Riseman, Frank R. Stolle Computer Science Department University of Massachusetts, USA Dong-Min Woo School of Electrical Engineering.

Image Based Rendering. Light Field Gershun in 1936 –An illuminated objects fills the surrounding space with light reflected of its surface, establishing.

Spring 2015 CSc 83020: 3D Photography Prof. Ioannis Stamos Mondays 4:15 – 6:15

Epitomic Location Recognition A generative approach for location recognition K. Ni, A. Kannan, A. Criminisi and J. Winn In proc. CVPR Anchorage,

112/5/ :54 Graphics II Image Based Rendering Session 11.

Yizhou Yu Texture-Mapping Real Scenes from Photographs Yizhou Yu Computer Science Division University of California at Berkeley Yizhou Yu Computer Science.

Photo VR Editor: A Panoramic and Spherical Environment Map Authoring Tool for Image-Based VR Browsers Jyh-Kuen Horng, Ming Ouhyoung Communications and.

CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai.

3-D Information cs5764: Information Visualization Chris North.

Image-Based Rendering Geometry and light interaction may be difficult and expensive to model –Think of how hard radiosity is –Imagine the complexity of.

Automatic 3D modelling of Architecture Anthony Dick 1 Phil Torr 2 Roberto Cipolla 1 1 Department of Engineering 2 Microsoft Research, University of Cambridge.

Presented by 翁丞世  View Interpolation  Layered Depth Images  Light Fields and Lumigraphs  Environment Mattes  Video-Based.

Jun Shimamura, Naokazu Yokoya, Haruo Takemura and Kazumasa Yamazawa

© 2005 University of Wisconsin

Image Based Modeling and Rendering (PI: Malik)

Closing Remarks.

Presentation transcript:

Three Dimensional Model Construction for Visualization Avideh Zakhor Video and Image Processing Lab University of California at Berkeley

Outline zGoals and objectives zPrevious work by PI zDirections for future work

Goals and Objectives zDevelop a framework for fast, automatic and accurate 3D model construction for objects, scenes, rooms, buildings (interior and exterior), urban areas, and cities. zModels must be easy to compute, compact to represent and suitable for high quality view synthesis and visualization zApplications: Virtual or augmented reality fly- throughs.

Previous Work on Scene Modeling zFull/Assisted 3-D Modeling Kanade et al.; Koch et al.; Becker & Bove; Debevec et al.; Faugeras et al.; Malik & Yu. zMosaics and Panoramas Szeliski & Kang; McMillan & Bishop; Shum & Szeliski zLayered/LDI Representations Wang & Adelson; Sawhney & Ayer; Weiss; Baker et al. zView Interpolation/IBR/Light Fields Chen & Williams; Chang & Zakhor; Laveau & Faugeras; Seitz & Dyer; Levoy & Hanrahan

Previous Work on Building Models zNevatia (USC): multi-sensor integration zTeller (MIT): spherical mosaics on a wheelchair sized rover, known 6DOF zVan Gool (Belgium): roof detection from aerial photographs zPeter Allen (Columbia): images and laser range finders; view/sensor planning. zFaugeras (INRIA)

Previous Work on City Modeling zPlanet 9: yCombines ground photographs with existing city maps manually. zUCLA Urban Simulation Team: yUses mutligen to create models from aerial photographs, together with ground video for texture mapping. zBath and London models by Univ. of Bath. yCombines aerial photgraphs with existing maps. zAll approaches are slow and labor intensive.

Work at VIP lab at UCB Scene modeling and reconstruction.

Multi-Valued Representation: MVR zLevel k has k occluding surfaces zForm multivalued array of depth and intensity

Observations

Imaging geometry (1) zPlanar translation

Imaging Geometry (2) zCircular/orbital motion

Dense Depth Estimation zEstimate camera motion zCompute depth maps to build MVRs y Low-contrast regions problematic for dense depth estimation. y Enforce spatial coherence to achieve realistic, high quality visualization.

Block Diagram for Dense Depth Estimation zPlanar approximation of depth for low contrast regions.

Oroginal Sequences “Mug” sequence (13 frames) “Teabox” sequence (102 frames)

Low-Contrast Regions Mug sequenceTea-box sequence zComplete tracking

Multiframe Depth Estimation Apply iterative estimation algorithm to enforce piecewise smoothness, without smoothing over depth discontinuities.

Multiframe Depth Estimation Multiframe Stereo + Low-Contrast Processing + Piecewise Smoothing Multiframe Stereo + Low-Contrast Processing + Piecewise Smoothing Mug Tea-box

Multivalued Representation zProject depths to reference coordinates

Results (1) zMug sequence Multivalued representation for frame 4 (Level 0)

Results zMug sequence Multivalued representation for frame 4 (Level 1)

Results zMug sequence Multivalued representation for frame 4 (Combining Levels 0 and 1)

Results Reconstructed sequenceArbitrary flythrough zMug sequence

Results (2) Multivalued representation for frame 22 (Intensity, Level 0) zTeabox sequence

Results Multivalued representation for frame 22 (Depth, Level 0) zTeabox sequence

Results Multivalued representation for frame 22 (Intensity, Level 1) zTeabox sequence

Results Multivalued representation for frame 22 (Depth, Level 1) zTeabox sequence

Results Multivalued representation for frame 22 (Intensity, combining Levels 0 and 1) zTeabox sequence

Results Multivalued representation for frame 22 (Depth, combining Levels 0 and 1) zTeabox sequence

Results Multivalued representation for frame 86 (Intensity, Level 0) zTeabox sequence

Results Multivalued representation for frame 86 (Depth, Level 0) zTeabox sequence

Results Multivalued representation for frame 86 (Intensity, Level 1) zTeabox sequence

Results Multivalued representation for frame 86 (Depth, Level 1) zTeabox sequence

Results Multivalued representation for frame 86 (Intensity, combining Levels 0 and 1) zTeabox sequence

Results Multivalued representation for frame 86 (Depth, combining Levels 0 and 1) zTeabox sequence

Multiple MVRs zPerform view interpolation w/many MVRs

Results: multiple MVRs Reconstructed sequence from MVR86 Reconstruct sequence from MVR22 zTeabox sequence

Results: Multiple MVRs Reconstructed sequenceArbitrary flyaround

Extensions zComplex scenes with many “levels” are difficult to model with MVR; e.g. trees, leaves, etc zDifficult to ensure realistic visualization from all angles; Need to plan capture process carefully. zTradeoff between CG polygon modeling and IBR; yUse both in real visualization databases. yBuild polygon models from MVR.

Issues for model construction zChoice of geometry for obtaining data zChoice of imaging technology. zChoice of representation. zChoice of models. zDealing with time varying scenes.

Extensions: zSo far, addressed “outside in” problem: yCamera looked inward to “scan” the object. zFuture work will focus on the “Inside out” problem: yModeling a room, office. yModeling exterior or interior of a building yModeling an urban environment e.g. a city

Strategy zUse: yRange sensors, position sensors (GPS), Gyros(orientation), omni camera, video. yExisting datasets: 3D CAD models, digital elevation maps (DEM), DTED, city maps, architectural drawings: apriori information

Modeling interior of buildings z Leverage existing work in the computer graphics group at UCB: y3D model of Soda hall available from the “soda walkthrough” project. y3D model built out of architectural drawings yUse additional video, and laser range finder input to xEnhance the details of the 3D model: furniture, etc xAdd texture maps for photo-realistic walk- throughs.

City Modeling zDevelop a framework for modeling parts of city of San Francisco: yUse aerial photograph as provided by Space Imaging Corp; resolution 1 ft. yUse digitized city maps yUse ground data collection vehicle to collect range and intensity video from a panoramic camera, annotated with 6 DOF parameters. yDerive data fusion algorithms to process the above in speedy, automated and accurate fashion.

Requirements zAutomation (little or no interaction needed from human operators) zSpeed: must scale with large areas and large data sets. zAccuracy zRobustness to location of data collection. zEase of data collection. zRepresentation suitable to hierarchical visualization databases.

Relationship to others zUSC: accurate tracking and registration algorithms needed for model construction. zSyracuse: uncertainty processing, and data fusion for model construction. zG. Tech: How to combine CG polygonal model building with IBR models in vis. database? How can vis. databases deal with photo-realistic rendering?

Conclusions zFast, accurate and automatic model construction is essential to mobile augmented reality systems. zOur goal is to provide photo-realistic rendering of objects, scenes, buildings, and cities, to enable, visualization, navigation and interaction.