1. CS395/495: Spring 2004 IBMR: Image Based Modeling and Rendering Introduction Jack Tumblin jet@cs.northwestern.edu

2. Admin: How this course works Refer to class website: (soon) http://www.cs.northwestern.edu/~jet/Teach/2004_ 3spr_IBMR/IBMRsyllabus2004.htmRefer to class website: (soon) http://www.cs.northwestern.edu/~jet/Teach/2004_ 3spr_IBMR/IBMRsyllabus2004.htm http://www.cs.northwestern.edu/~jet/Teach/2004_ 3spr_IBMR/IBMRsyllabus2004.htm http://www.cs.northwestern.edu/~jet/Teach/2004_ 3spr_IBMR/IBMRsyllabus2004.htm Tasks:Tasks: –Reading, lectures, class participation, projects Evaluation:Evaluation: –Progressive Programming Project –Take-Home Midterm –In-class project demo; no final exam

3. GOAL: First-Class Primitive Want images as ‘first-class’ primitivesWant images as ‘first-class’ primitives –Useful as BOTH input and output –Convert to/from traditional scene descriptions Want to mix real & synthetic scenes freelyWant to mix real & synthetic scenes freely Want to extend photographyWant to extend photography –Easily capture scene: shape, movement, surface/BRDF, lighting … –Modify & Render the captured scene data --BUT----BUT-- images hold only PARTIAL scene informationimages hold only PARTIAL scene information “You can’t always get what you want” “You can’t always get what you want” –(Mick Jagger 1968)

4. Back To Basics: Scene & Image Light + 3D Scene: Illumination, shape, movement, surface BRDF,… Image Plane I(x,y) Angle( ,  ) Position(x,y) 2D Image: Collection of rays through a point

5. Trad. Computer Graphics Image Plane I(x,y) Angle( ,  ) Position(x,y) 2D Image: Collection of rays through a point Light + 3D Scene: Illumination, shape, movement, surface BRDF,… Reduced,IncompleteInformation

6. !TOUGH!‘ILL-POSED’ Many Simplifications, External knowledge… Trad. Computer Vision Image Plane I(x,y) Angle( ,  ) Position(x,y) 2D Image: Collection of rays through a point Light + 3D Scene: Light + 3D Scene: Illumination, shape, movement, surface BRDF,…

7. 2D Display Image(s) IBMR Goal: Bidirectional Rendering Both forward and ‘inverse’ rendering!Both forward and ‘inverse’ rendering! Camera Pose Camera View Geom Scene illumination Object shape, position Surface reflectance,… transparency 2D Display Image(s) IBMR TraditionalComputerGraphics ‘3D Scene’ Description ‘Optical’Description ?! New Research?

8. OLDEST IBR: Shadow Maps (1984) Fast Shadows from Z-buffer hardware: 1) Make the “Shadow Map”: –Render image seen from light source, BUT –Keep ONLY the Z-buffer values (depth) 2) Render Scene from Eyepoint: –Pixel + Z depth gives 3D position of surface; –Project 3D position into Shadow map image –If Shadow Map depth < 3D depth, SHADOW!

9. Early IBR: QuickTime VR (Chen, Williams ’93) 1) Four Planar Images  1 Cylindrical Panorama: Re-sampling Required! Planar Pixels: equal distance on x,y plane (tan -1  ) Cylinder Pixs: horiz: equal angle on cylinder (  ) vert: equal distance on y (tan -1  ) Cylinder Pixs: horiz: equal angle on cylinder (  ) vert: equal distance on y (tan -1  )

10. Early IBR: QuickTime VR (Chen, Williams ’93) 1) Four Planar Images  1 Cylindrical Panorama: IN:OUT:

11. Early IBR: QuickTime VR (Chen, Williams ’93) 2) Windowing, Horizontal-only Reprojection: IN:OUT:

12. Early IBR: QuickTime VR (Chen, Williams ’93) 2) Windowing, Horizontal-only Reprojection: IN:OUT:

13. View Interpolation: How? But what if no depth is available?But what if no depth is available? Traditional Stereo Disparity Map: pixel-by-pixel search for correspondenceTraditional Stereo Disparity Map: pixel-by-pixel search for correspondence

14. View Interpolation: How? Store Depth at each pixel: reprojectStore Depth at each pixel: reproject Coarse or Simple 3D model:Coarse or Simple 3D model:

15. Plenoptic Array: ‘The Matrix Effect’ Brute force! Simple arc, line, or ring array of camerasBrute force! Simple arc, line, or ring array of cameras Synchronized shutter http://www.ruffy.com/firingline.htmlSynchronized shutter http://www.ruffy.com/firingline.html http://www.ruffy.com/firingline.html Warp/blend between images to change viewpoint on ‘time-frozen’ scene:Warp/blend between images to change viewpoint on ‘time-frozen’ scene:

16. for a given scene, describe ALL rays throughfor a given scene, describe ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, –ALL time F(x,y,z, , ,, t) “Eyeballs Everywhere” function (5-D x 2-D!) Plenoptic Function (Adelson, Bergen `91) … … … … … … … … … ……

17. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm 1)Manually set some corresp.points (eye corners, etc.) 2) pre-warp and post-warp to match points in 3D, 3) Reproject for Virtual cameras

18. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

19. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

20. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

21. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

22. ‘Scene’ causes Light Field Light field: holds all outgoing light rays Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light … Cameras capture subset of these rays.

23. ? Can we recover Shape ? Can you find ray intersections? Or ray depth? Ray colors might not match for non-diffuse materials (BRDF)

24. ? Can we recover Surface Material ? Can you find ray intersections? Or ray depth? Ray colors might not match for non-diffuse materials (BRDF)

25. Hey, wait… Light field describes light LEAVING the enclosing surface….Light field describes light LEAVING the enclosing surface…. ? Isn’t there a complementary ‘light field’ for the light ENTERING the surface?? Isn’t there a complementary ‘light field’ for the light ENTERING the surface? *YES* ‘Image-Based Lighting’ too!*YES* ‘Image-Based Lighting’ too!

26. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera

27. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera cccc cccc

28. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera Projector (laser brick)

29. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene –(and wavelength and time) F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera projector

30. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene –(and wavelength and time) F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera projector

31. ! Complete !! Complete ! –Geometry, Lighting, BRDF,… ! NOT REQUIRED ! Preposterously Huge: (?!?! 8-D function sampled at image resolution !?!?), butPreposterously Huge: (?!?! 8-D function sampled at image resolution !?!?), but Hugely Redundant:Hugely Redundant: –Wavelength  RGB triplets, ignore Time, and Restrict eyepoint movement: maybe ~3 or 4D ? –Very Similar images—use Warping rules? (SIGG2002) –Exploit Movie Storage/Compression Methods?

32. IBR-Motivating Opinions “Computer Graphics: Hard” –Complex! geometry, texture, lighting, shadows, compositing, BRDF, interreflections, etc. etc., etc., … –Irregular! Visibility,Topology, Render Eqn.,… –Isolated! Tough to use real objects in CGI –Slow! compute-bound, off-line only,… “Digital Imaging: Easy” –Simple! More quality? Just pump more pixels! –Regular! Vectorized, compressible, pipelined… –Accessible! Use real OR synthetic (CGI) images! –Fast! Scalable, Image reuse, a path to interactivity…

33. Practical IBMR What useful partial solutions are possible? Texture Maps++: Panoramas, Env. MapsTexture Maps++: Panoramas, Env. Maps Image(s)+Depth: (3D shell)Image(s)+Depth: (3D shell) Estimating Depth & Recovering SilhouettesEstimating Depth & Recovering Silhouettes ‘Light Probe’ measures real-world light‘Light Probe’ measures real-world light Structured Light to Recover Surfaces…Structured Light to Recover Surfaces… Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

34. Conclusion Heavy overlap with computer vision: careful not to re-invent & re-name!Heavy overlap with computer vision: careful not to re-invent & re-name! Elegant Geometry is at the heart of it all, even surface reflectance, illumination, etc. etc.Elegant Geometry is at the heart of it all, even surface reflectance, illumination, etc. etc. THUS: we’ll dive into geometry--all the rest is built on it!THUS: we’ll dive into geometry--all the rest is built on it!