1. Spring 2003 IBMR: Image Based Modeling and Rendering Jack Tumblin Computer Science Dept., Northwestern University jet@cs.northwestern.edu

2. Pat Hanrahan, 1999 Trompe L’oeil Examples (Palo Alto CA Trompe L’oeil Examples (Palo Alto CA )

3. 2D Display Image(s) IBMR Goal: Bidirectional Rendering Both forward and ‘inverse’ rendering!Both forward and ‘inverse’ rendering! Camera Pose Camera View Geom Scene illumination Object shape, position Surface reflectance,… transparency 2D Display Image(s) IBMR TraditionalComputerGraphics ‘3D Scene’ Description ‘Optical’Description ?! New Research?

4. IBMR Apps : Why Bother? Movie Special FX: Mix real & syntheticMovie Special FX: Mix real & synthetic Augmented Photography/Video:Augmented Photography/Video: –Any Camera Position, –Any Lighting, –Any moment in Time Historical Preservation/RestorationHistorical Preservation/Restoration The Persistent, Tantalizing CG/Vision Failures: Digital/Visual Libraries, 3D Mo-Cap, VR, haptics, telepresence, Avatar I/O, Facial ID,...The Persistent, Tantalizing CG/Vision Failures: Digital/Visual Libraries, 3D Mo-Cap, VR, haptics, telepresence, Avatar I/O, Facial ID,...

5. Synthetic Image + Depth = Surface? “Hole-filling”Problem;(incomplete,uneven surface data) ShadingProblem;need reflectance w/o lighting,... McMillan 1996

6. Do More pictures (or photos) Help?

7. Does Motion & Video Help? time  J. Shade et al., 1998

8. *OR* Can Images Help Rendering? “Billboards” in the distance, or “impostors” instead of complex geometry? -Sillion et. al 1999

9. “Image-Based Shadows” Agrawala2000 Yes, For some limited but very tough problems...

10. “Deep Shadow Maps” Lokovic2000 Yes, For some limited but very tough problems...

11. Early IBR: QuickTime VR (Chen, Williams ’93) Four Planar Images  1 Cylindrical Panorama User-steered, real-time view in any direction

12. Early IBR: QuickTime VR (Chen, Williams ’93) 2) Windowing, Horizontal-only Reprojection: IN:OUT:

13. View Interpolation: How? But how can we DISPLACE the camera?But how can we DISPLACE the camera? What if no depth is available?What if no depth is available? Traditional Stereo Disparity Map: pixel-by-pixel search for correspondence (an unreliable mess)Traditional Stereo Disparity Map: pixel-by-pixel search for correspondence (an unreliable mess)

14. Plenoptic Array: ‘The Matrix Effect’ Brute force! Simple arc, line, or ring array of camerasBrute force! Simple arc, line, or ring array of cameras Synchronized shutter http://www.ruffy.com/firingline.htmlSynchronized shutter http://www.ruffy.com/firingline.html http://www.ruffy.com/firingline.html Warp/blend between images to change viewpoint on ‘time-frozen’ scene:Warp/blend between images to change viewpoint on ‘time-frozen’ scene:

15. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm 1)Manually set some corresp.points (eye corners, etc.) 2) pre-warp and post-warp to match points in 3D, 3) Reproject for Virtual cameras A Mostly 2-D Method

16. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm A Mostly 2-D Method

17. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm A Mostly 2-D Method

18. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm A Mostly 2-D Method

19. Seitz: ‘View Morphing’ SIGG`96 http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm A Mostly 2-D Method

20. Malzbender, HPlabs 2001 A Mostly 2-D Method Polynomial Texture Maps Store just 6 coefficients at each pixel, get Interactive re-lighting...

21. for a given scene, describe ALL rays throughfor a given scene, describe ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, –ALL time F(x,y,z, , ,, t) “Eyeballs Everywhere” function (5-D x 2-D!) 5-D Plenoptic Function (Adelson, Bergen `91) … … … … … … … … … ……

22. ‘Scene’ causes 4D Light Field Light field: holds all outgoing light rays Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light … Cameras capture subset of these rays.

23. Levoy et al., “Light Fields” 1996

24. ? Can we recover Shape ? Can you find ray intersections? Or ray depth? Ray colors might not match for shiny materials (BRDF)

25. ? Can we recover Surface Material ? Can you find ray intersections? Or ray depth? Ray colors might not match for shiny materials (BRDF)

26. Hey, wait… ‘Light field’ describes light LEAVING the enclosing surface….‘Light field’ describes light LEAVING the enclosing surface…. ? Isn’t there a complementary ‘light field’ for the light ENTERING the surface?? Isn’t there a complementary ‘light field’ for the light ENTERING the surface? *YES*  ‘Image-Based Lighting’ too!*YES*  ‘Image-Based Lighting’ too! ? How can we capture incoming light?? How can we capture incoming light?

27. A Mirror Sphere is... A panoramic camera converter, and A panoramic camera converter, and

28. A ‘light probe’ to measure ALL incoming light at a point. How can we use it? A Mirror Sphere is...

29. Image-Based Actual Re-lighting

30. Film the background in Milan, Measure incoming light, Light the actress in Los Angeles Matte the background Matched LA and Milan lighting. Debevec et al., SIGG2001

31. Measure REAL light in a REAL scene... Debevec et al., SIGG1998

32. Render FAKE objects with REAL light, Debevec et al., SIGG1998 And combine with REAL image:

33. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera

34. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera cccc cccc

35. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera Projector (laser brick)

36. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene –(and wavelength and time) F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera projector

37. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene –(and wavelength and time) F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera projector

38. ! Complete !! Complete ! –Geometry, Lighting, BRDF,…it’s all there, –It’s all LINEAR, –But it’s an 8-D function! Stupendously Huge! (40GB data sets, 32 hours / object…) What Subsets Are Useful?What Subsets Are Useful? –Fixed camera, vary lighting? compositing? –Fixed lighting, vary camera? animation? –Recover some shape? silhouettes? –Recover some reflectance? –Recover some new hybrid?

39. Image-Based Synthetic Re-lighting Masselus et al., 2002

40. Basis images Illuminant direction estimation Take pictures and move hand-held light source Incident light map Create weighted sum … · W 1 + · W 2 + + · W n Relit object Masselus et al., 2002, “Free-Form Light Stage”

41. Rushmeier, 2001 Fine geometric Details  Fine Texture/Normal Details Image-Based Shape Refinement

42. Matusik 2002 Image-Based Shape Approximation

43. Matusik 2002 Image-Based Shape Approximation

44. Practical IBMR (so far...) What useful partial solutions are possible? Texture Maps++: Panoramas, Env. MapsTexture Maps++: Panoramas, Env. Maps Image(s)+Depth: (3D shell)Image(s)+Depth: (3D shell) Estimating Depth & Recovering SilhouettesEstimating Depth & Recovering Silhouettes ‘Light Probe’ measures real-world light‘Light Probe’ measures real-world light Structured Light to Recover Surfaces…Structured Light to Recover Surfaces… Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

45. IBMR Challenges Bulky Data; High-Dimensionality!Bulky Data; High-Dimensionality! Representation, Compression, Fast accessRepresentation, Compression, Fast access Editing, Control, FlexibilityEditing, Control, Flexibility Self-movement, Animation ?Self-movement, Animation ? Complex interreflections, scattering,...Complex interreflections, scattering,... Compositing, Self-occlusion, lighting/view pose,Compositing, Self-occlusion, lighting/view pose, concavities, geometry/detail, resolution...concavities, geometry/detail, resolution...

46. Conclusion Heavy overlap with computer vision: careful not to re-invent & re-name!Heavy overlap with computer vision: careful not to re-invent & re-name! Elegant Geometry is at the heart of it all, even surface reflectance, illumination, etc. etc. 8-10 dimensional, but linear functions.Elegant Geometry is at the heart of it all, even surface reflectance, illumination, etc. etc. 8-10 dimensional, but linear functions. Melting barrier between real, synthetic 3D.Melting barrier between real, synthetic 3D.

47. IBMR: Visiting Lecturers 3D Scanning for Cultural Heritage Applications Holly Rushmeier, IBM TJ Watson Friday May 16 3:00pm, Rm 381, CS Dept. ?Subsurface Scattering? ?Rendering Human Hair? Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.

48. END