Spring 2003 IBMR: Image Based Modeling and Rendering Jack Tumblin Computer Science Dept., Northwestern University

Slides:



Advertisements
Similar presentations
Computer Graphics Inf4/MSc Computer Graphics Lecture Notes #16 Image-Based Modelling, Rendering and Lighting.
Advertisements

Lecture 35: Photometric stereo CS4670 / 5670 : Computer Vision Noah Snavely.
CS395/495: Spring 2004 IBMR: Image Based Modeling and Rendering Introduction Jack Tumblin
Introduction to Image-Based Rendering Jian Huang, CS 594, Spring 2002 A part of this set of slides reference slides used at Standford by Prof. Pat Hanrahan.
Measuring BRDFs. Why bother modeling BRDFs? Why not directly measure BRDFs? True knowledge of surface properties Accurate models for graphics.
Light Fields PROPERTIES AND APPLICATIONS. Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene 
Acquiring the Reflectance Field of a Human Face Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, Mark Sagar Haarm-Pieter Duiker,
Lecture 23: Photometric Stereo CS4670/5760: Computer Vision Kavita Bala Scott Wehrwein.
Advanced Computer Graphics CSE 190 [Spring 2015], Lecture 14 Ravi Ramamoorthi
Copyright  Philipp Slusallek Cs fall IBR: Model-based Methods Philipp Slusallek.
CS 395: Adv. Computer Graphics Overview: Image-Based Modeling and Rendering Jack Tumblin
CS 395: Adv. Computer Graphics Image-Based Modeling and Rendering Jack Tumblin
Light Readings Forsyth, Chapters 4, 6 (through 6.2) by Ted Adelson.
Advanced Computer Graphics (Fall 2010) CS 283, Lecture 16: Image-Based Rendering and Light Fields Ravi Ramamoorthi
Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi
Representations of Visual Appearance COMS 6160 [Spring 2007], Lecture 4 Image-Based Modeling and Rendering
Image-Based Modeling and Rendering CS 6998 Lecture 6.
Image-Based Lighting : Computational Photography Alexei Efros, CMU, Fall 2005 © Eirik Holmøyvik …with a lot of slides donated by Paul Debevec.
View interpolation from a single view 1. Render object 2. Convert Z-buffer to range image 3. Re-render from new viewpoint 4. Use depths to resolve overlaps.
Representations of Visual Appearance COMS 6160 [Spring 2007], Lecture 3 Ravi Ramamoorthi
Image-based Rendering of Real Objects with Complex BRDFs.
3D from multiple views : Rendering and Image Processing Alexei Efros …with a lot of slides stolen from Steve Seitz and Jianbo Shi.
I hope that you: Looked at book & website Checked Pre-requisites (change before Friday!) Participate! Ask Questions! Get Inspired … CS395: Advanced Computer.
CS 395/495-25: Spring 2003 IBMR: Week 10A Dimensionality of IBMR Methods Jack Tumblin
Image Stitching and Panoramas
Copyright  Philipp Slusallek IBR: View Interpolation Philipp Slusallek.
CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai.
1Jana Kosecka, CS 223b Cylindrical panoramas Cylindrical panoramas with some slides from R. Szeliski, S. Seitz, D. Lowe, A. Efros,
Siggraph’2000, July 27, 2000 Jin-Xiang Chai Xin Tong Shing-Chow Chan Heung-Yeung Shum Microsoft Research, China Plenoptic Sampling SIGGRAPH’2000.
CSCE 641: Computer Graphics Image-based Rendering Jinxiang Chai.
NVIDIA Lecture 10 Copyright  Pat Hanrahan Image-Based Rendering: 1st Wave Definition: Using images to enhance the realism of 3D graphics Brute Force in.
CS 395/495-25: Spring 2003 IBMR: Week 9A Image-Based Physics: Measuring Light & Materials Jack Tumblin
Stereo Guest Lecture by Li Zhang
Project 1 artifact winners Project 2 questions Project 2 extra signup slots –Can take a second slot if you’d like Announcements.
 Marc Levoy IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry.”
 Marc Levoy IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry.”
View interpolation from a single view 1. Render object 2. Convert Z-buffer to range image 3. Re-render from new viewpoint 4. Use depths to resolve overlaps.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
Measure, measure, measure: BRDF, BTF, Light Fields Lecture #6
Computer Graphics Inf4/MSc Computer Graphics Lecture Notes #16 Image-Based Lighting.
Real-Time High Quality Rendering CSE 291 [Winter 2015], Lecture 6 Image-Based Rendering and Light Fields
Announcements Project 1 artifact winners Project 2 questions
Advanced Computer Graphics (Spring 2013) CS 283, Lecture 15: Image-Based Rendering and Light Fields Ravi Ramamoorthi
The Free-form Light Stage Vincent Masselus Philip Dutré Frederik Anrys Department of Computer Science.
Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH.
11/15/11 Image-based Lighting Computational Photography Derek Hoiem, University of Illinois Many slides from Debevec, some from Efros T2.
Image-based rendering Michael F. Cohen Microsoft Research.
03/12/03© 2003 University of Wisconsin Last Time NPR Assignment Projects High-Dynamic Range Capture Image Based Rendering Intro.
. Wild Dreams for Cameras Jack Tumblin Northwestern University From May 24 Panel Discussion on cameras.
Image-based Rendering. © 2002 James K. Hahn2 Image-based Rendering Usually based on 2-D imagesUsually based on 2-D images Pre-calculationPre-calculation.
CS 395: Adv. Computer Graphics Light Fields and their Approximations Jack Tumblin
Image Based Rendering. Light Field Gershun in 1936 –An illuminated objects fills the surrounding space with light reflected of its surface, establishing.
Computing & Information Sciences Kansas State University Lecture 10 of 42CIS 636/736: (Introduction to) Computer Graphics CIS 636/736 Computer Graphics.
Image-Based Lighting © Eirik Holmøyvik …with a lot of slides donated by Paul Debevec CS194: Image Manipulation & Computational Photography Alexei Efros,
What is light? Electromagnetic radiation (EMR) moving along rays in space R( ) is EMR, measured in units of power (watts) – is wavelength Light: Travels.
Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography.
CSL 859: Advanced Computer Graphics Dept of Computer Sc. & Engg. IIT Delhi.
Interreflections : The Inverse Problem Lecture #12 Thanks to Shree Nayar, Seitz et al, Levoy et al, David Kriegman.
Announcements Office hours today 2:30-3:30 Graded midterms will be returned at the end of the class.
112/5/ :54 Graphics II Image Based Rendering Session 11.
Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.
CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai.
Don Norman’s Fall 2003 Seminar 8-Dimensional Digital Photography for Historical Preservation Jack Tumblin
Image-Based Rendering Geometry and light interaction may be difficult and expensive to model –Think of how hard radiosity is –Imagine the complexity of.
Sub-Surface Scattering Real-time Rendering Sub-Surface Scattering CSE 781 Prof. Roger Crawfis.
CS 395: Adv. Computer Graphics Light Fields and their Approximations Jack Tumblin
Advanced Computer Graphics
Image-based Lighting Computational Photography
Image-Based Rendering
© 2005 University of Wisconsin
Presentation transcript:

Spring 2003 IBMR: Image Based Modeling and Rendering Jack Tumblin Computer Science Dept., Northwestern University

Pat Hanrahan, 1999 Trompe L’oeil Examples (Palo Alto CA Trompe L’oeil Examples (Palo Alto CA )

2D Display Image(s) IBMR Goal: Bidirectional Rendering Both forward and ‘inverse’ rendering!Both forward and ‘inverse’ rendering! Camera Pose Camera View Geom Scene illumination Object shape, position Surface reflectance,… transparency 2D Display Image(s) IBMR TraditionalComputerGraphics ‘3D Scene’ Description ‘Optical’Description ?! New Research?

IBMR Apps : Why Bother? Movie Special FX: Mix real & syntheticMovie Special FX: Mix real & synthetic Augmented Photography/Video:Augmented Photography/Video: –Any Camera Position, –Any Lighting, –Any moment in Time Historical Preservation/RestorationHistorical Preservation/Restoration The Persistent, Tantalizing CG/Vision Failures: Digital/Visual Libraries, 3D Mo-Cap, VR, haptics, telepresence, Avatar I/O, Facial ID,...The Persistent, Tantalizing CG/Vision Failures: Digital/Visual Libraries, 3D Mo-Cap, VR, haptics, telepresence, Avatar I/O, Facial ID,...

Synthetic Image + Depth = Surface? “Hole-filling”Problem;(incomplete,uneven surface data) ShadingProblem;need reflectance w/o lighting,... McMillan 1996

Do More pictures (or photos) Help?

Does Motion & Video Help? time  J. Shade et al., 1998

*OR* Can Images Help Rendering? “Billboards” in the distance, or “impostors” instead of complex geometry? -Sillion et. al 1999

“Image-Based Shadows” Agrawala2000 Yes, For some limited but very tough problems...

“Deep Shadow Maps” Lokovic2000 Yes, For some limited but very tough problems...

Early IBR: QuickTime VR (Chen, Williams ’93) Four Planar Images  1 Cylindrical Panorama User-steered, real-time view in any direction

Early IBR: QuickTime VR (Chen, Williams ’93) 2) Windowing, Horizontal-only Reprojection: IN:OUT:

View Interpolation: How? But how can we DISPLACE the camera?But how can we DISPLACE the camera? What if no depth is available?What if no depth is available? Traditional Stereo Disparity Map: pixel-by-pixel search for correspondence (an unreliable mess)Traditional Stereo Disparity Map: pixel-by-pixel search for correspondence (an unreliable mess)

Plenoptic Array: ‘The Matrix Effect’ Brute force! Simple arc, line, or ring array of camerasBrute force! Simple arc, line, or ring array of cameras Synchronized shutter shutter Warp/blend between images to change viewpoint on ‘time-frozen’ scene:Warp/blend between images to change viewpoint on ‘time-frozen’ scene:

Seitz: ‘View Morphing’ SIGG` )Manually set some corresp.points (eye corners, etc.) 2) pre-warp and post-warp to match points in 3D, 3) Reproject for Virtual cameras A Mostly 2-D Method

Seitz: ‘View Morphing’ SIGG`96 A Mostly 2-D Method

Seitz: ‘View Morphing’ SIGG`96 A Mostly 2-D Method

Seitz: ‘View Morphing’ SIGG`96 A Mostly 2-D Method

Seitz: ‘View Morphing’ SIGG`96 A Mostly 2-D Method

Malzbender, HPlabs 2001 A Mostly 2-D Method Polynomial Texture Maps Store just 6 coefficients at each pixel, get Interactive re-lighting...

for a given scene, describe ALL rays throughfor a given scene, describe ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, –ALL time F(x,y,z, , ,, t) “Eyeballs Everywhere” function (5-D x 2-D!) 5-D Plenoptic Function (Adelson, Bergen `91) … … … … … … … … … ……

‘Scene’ causes 4D Light Field Light field: holds all outgoing light rays Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light … Cameras capture subset of these rays.

Levoy et al., “Light Fields” 1996

? Can we recover Shape ? Can you find ray intersections? Or ray depth? Ray colors might not match for shiny materials (BRDF)

? Can we recover Surface Material ? Can you find ray intersections? Or ray depth? Ray colors might not match for shiny materials (BRDF)

Hey, wait… ‘Light field’ describes light LEAVING the enclosing surface….‘Light field’ describes light LEAVING the enclosing surface…. ? Isn’t there a complementary ‘light field’ for the light ENTERING the surface?? Isn’t there a complementary ‘light field’ for the light ENTERING the surface? *YES*  ‘Image-Based Lighting’ too!*YES*  ‘Image-Based Lighting’ too! ? How can we capture incoming light?? How can we capture incoming light?

A Mirror Sphere is... A panoramic camera converter, and A panoramic camera converter, and

A ‘light probe’ to measure ALL incoming light at a point. How can we use it? A Mirror Sphere is...

Image-Based Actual Re-lighting

Film the background in Milan, Measure incoming light, Light the actress in Los Angeles Matte the background Matched LA and Milan lighting. Debevec et al., SIGG2001

Measure REAL light in a REAL scene... Debevec et al., SIGG1998

Render FAKE objects with REAL light, Debevec et al., SIGG1998 And combine with REAL image:

Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera

Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera cccc cccc

Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera Projector (laser brick)

Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene –(and wavelength and time) F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera projector

Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene –(and wavelength and time) F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera projector

! Complete !! Complete ! –Geometry, Lighting, BRDF,…it’s all there, –It’s all LINEAR, –But it’s an 8-D function! Stupendously Huge! (40GB data sets, 32 hours / object…) What Subsets Are Useful?What Subsets Are Useful? –Fixed camera, vary lighting? compositing? –Fixed lighting, vary camera? animation? –Recover some shape? silhouettes? –Recover some reflectance? –Recover some new hybrid?

Image-Based Synthetic Re-lighting Masselus et al., 2002

Basis images Illuminant direction estimation Take pictures and move hand-held light source Incident light map Create weighted sum … · W 1 + · W · W n Relit object Masselus et al., 2002, “Free-Form Light Stage”

Rushmeier, 2001 Fine geometric Details  Fine Texture/Normal Details Image-Based Shape Refinement

Matusik 2002 Image-Based Shape Approximation

Matusik 2002 Image-Based Shape Approximation

Practical IBMR (so far...) What useful partial solutions are possible? Texture Maps++: Panoramas, Env. MapsTexture Maps++: Panoramas, Env. Maps Image(s)+Depth: (3D shell)Image(s)+Depth: (3D shell) Estimating Depth & Recovering SilhouettesEstimating Depth & Recovering Silhouettes ‘Light Probe’ measures real-world light‘Light Probe’ measures real-world light Structured Light to Recover Surfaces…Structured Light to Recover Surfaces… Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

IBMR Challenges Bulky Data; High-Dimensionality!Bulky Data; High-Dimensionality! Representation, Compression, Fast accessRepresentation, Compression, Fast access Editing, Control, FlexibilityEditing, Control, Flexibility Self-movement, Animation ?Self-movement, Animation ? Complex interreflections, scattering,...Complex interreflections, scattering,... Compositing, Self-occlusion, lighting/view pose,Compositing, Self-occlusion, lighting/view pose, concavities, geometry/detail, resolution...concavities, geometry/detail, resolution...

Conclusion Heavy overlap with computer vision: careful not to re-invent & re-name!Heavy overlap with computer vision: careful not to re-invent & re-name! Elegant Geometry is at the heart of it all, even surface reflectance, illumination, etc. etc dimensional, but linear functions.Elegant Geometry is at the heart of it all, even surface reflectance, illumination, etc. etc dimensional, but linear functions. Melting barrier between real, synthetic 3D.Melting barrier between real, synthetic 3D.

IBMR: Visiting Lecturers 3D Scanning for Cultural Heritage Applications Holly Rushmeier, IBM TJ Watson Friday May 16 3:00pm, Rm 381, CS Dept. ?Subsurface Scattering? ?Rendering Human Hair? Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.

END