Spring 2015 CSc 83020: 3D Photography Prof. Ioannis Stamos Mondays 4:15 – 6:15

3D Photography > complex / accurate representation of a 3D scene

Overview Create geometric and photometric 3D models Use Range and Image Sensing Fusing image data Comprehensive system with automation

Computer Vision Physical 3D World Illumination Vision System Scene Description Sensors: Digital Cameras (2D) Range Scanners (3D) GeometryReflectance

3D Photography & Graphics Model of the Physical World Model of Illumination Scene Description GeometryReflectance Modeling [Representation of 3D objects] Rendering [Construction of 2D images from 3D models] Animation [Simulating changes over time] Captured from Vision System

3D PHOTOGRAPHY EXAMPLE Ten scans were acquired of façade of Thomas Hunter Building (NYC) Registration details Automatic registration. Each scan has a different color.

3D PHOTOGRAPHY EXAMPLE 24 scans were acquired of façade of Shepard Hall (City College of NY)

DataAcquisition Leica ScanStation 2 Spherical field of view. Registered color camera. Max Range: 300 meters. Scanning time: 2-3 times faster Accuracy: ~5mm per range point

Data Acquisition, Leica Scan Station 2, Park Avenue and 70 th Street, NY

Applications Virtual environment generation – Google Earth – acquire model for use in VRML, entertainment, etc – Realistic sets: movies and video games Reverse engineering – acquiring a model from a part copying/modification Part inspection – compare acquired model to “acceptable” model 3D FAX – transmit acquired model to remote RP machine Architectural site modeling Urban Planning Historical Preservation and Archaeology Reverse Engineering of Buildings

Data Acquisition Example Color Image of Thomas Hunter Building, New York City. Range Image. One million 3D points. Pseudocolor corresponds to laser intensity.

Traditional Pipeline Range Images Segmentation 3D feature extraction Partial Model Complete Model Range-Range Registration Brightness Images 2D feature extraction Range-Intensity Registration FINAL MODEL FINAL PHOTOREALISTIC MODEL

Other range sensors (some based on PrimeSense/Kinect) Matterport DotProduct Google Project Tango Prototype

Course Format Instructor will present recent topics and necessary background material Each student will present one or two papers Two projects Final grade: 50% : projects 30% : presentation 20% : class participation

Major Topics Acquiring images: 2D and 3D sensors (digital cameras and laser range scanners). 3D- and 2D-image registration. Geometry: – representation of 3D models, – simplification of 3D models, – detection of symmetry. Classification. Rendering 3D models. Image based rendering. Texture mapping.

Stereo Vision depth map

Segmentation

REGISTRATION pairwise & global

3D Modeling (Mesh or volumetric)

Model simplification

Passive techniques: Stereo and Structure from Motion

3D range to 2D image registration 3D scene 2D image

Texture mapped 3D model Corresponding 2D/proj. 3D lines 3D range to 2D image registration

TEXTURE MAP ANIMATION

The Façade Modeling System

Symmetry Detection

Image-Based Rendering Chen and Williams (1993) - view interpolation Chen and Williams (1993) - view interpolation McMillan and Bishop (1995) - plenoptic modeling McMillan and Bishop (1995) - plenoptic modeling Levoy and Hanrahan (1996) - light field rendering Levoy and Hanrahan (1996) - light field rendering Slide by Ravi Ramamoorthi, Columbia University

modelinganimationrendering 3D scanning motion capture image-based rendering The graphics pipeline the traditional pipeline the new pipeline? Slide courtesy Marc Levoy

Dynamic Scenes Image sequence (CMU, Virtualized Reality Project)

Dynamic Scenes Dynamic 3D model (CMU, Virtualized Reality Project)

Dynamic Scenes Dynamic texture-mapped model (CMU, Virtualized Reality Project)

Libraries Open Inventor Graphics Libraries Coin3D implements Open Inventor API: Online book: Book: The Inventor Mentor : Programming Object- Oriented 3D Graphics with Open Inventor, Release 2

33 Experimental Results Details of merged segments Merged points Surface meshes

Ball Pivoting Algorithm F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, G. Taubin. The ball-pivoting algorithm for surface reconstruction. IEEE Trans. on Vis. and Comp. Graph. 5 (4), pages , October-December A sequence of ball-pivoting operations. From left to right: A seed triangle is found; pivoting around an edge of the current front adds a new triangle to the mesh; after a number of pivoting operations, the active front closes on itself; a final ball-pivoting completes the mesh. Closely related to alpha-shapes, Edelsbrunner 94

Ball Pivoting Algorithm Results 3D mesh detail

RECOVERED CAMERA POSITIONS

GRAND CENTRAL (4)

GRAND CENTRAL (5)

Image-Based Rendering Image-Based Rendering – Chen and Williams (1993) - view interpolation – McMillan and Bishop (1995) - plenoptic modeling – Levoy and Hanrahan (1996) - light field rendering – Ng and Levoy (2005) – light field photography -> lytro camera – Slide by Ravi Ramamoorthi, Columbia University

3D Modeling (people) Ioannis Stamos – CSCI F08 Slide courtesy of Sebastian Thrun Stanford

Head of Michelangelo’s David photograph1.0 mm computer model