1. CS 395: Adv. Computer Graphics Image-Based Modeling and Rendering Jack Tumblin jet@cs.northwestern.edu

2. GOAL: First-Class Primitive Want images as ‘first-class’ primitivesWant images as ‘first-class’ primitives –Useful as BOTH input and output –Convert to/from traditional scene descriptions Want to mix real & synthetic scenes freelyWant to mix real & synthetic scenes freely Want to extend photographyWant to extend photography –Easily capture scene: shape, movement, surface/BRDF, lighting … –Modify & Render the captured scene data “You can’t always get what you want”“You can’t always get what you want” –(Mick Jagger 1968)

3. Back To Basics: Scene & Image Light + 3D Scene: Illumination, shape, movement, surface BRDF,… Image Plane I(x,y) Angle( ,  ) Position(x,y) 2D Image: Collection of rays through a point

4. Trad. Computer Graphics Image Plane I(x,y) Angle( ,  ) Position(x,y) 2D Image: Collection of rays through a point Light + 3D Scene: Illumination, shape, movement, surface BRDF,… Reduced,IncompleteInformation

5. !TOUGH!‘ILL-POSED’ Many Simplifications, External knowledge… Trad. Computer Vision Image Plane I(x,y) Angle( ,  ) Position(x,y) 2D Image: Collection of rays through a point Light + 3D Scene: Light + 3D Scene: Illumination, shape, movement, surface BRDF,…

6. for a given scene, describe:for a given scene, describe: –ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, –ALL time F(x,y,z, , ,, t) “Eyeballs Everywhere” function (7-D!) Plenoptic Function (Adelson, Bergen `91) … … … … … … … … … ……

7. A Big Plenoptic Question: Image entraps a partial scene description,… –Computer Vision problem: 3D->2D –Image point  scene surface point (usually) –Occlusion hides some scene surfaces –(BRDF * irradiance) tough to split apart! ? Does Plenoptic fcn. contain full scene ? –Exhaustive record of all image rays –Even SIMPLEST scene is huge, redundant, –The ‘consequences’ of all possible renderings* so

8. ‘Scene’ causes Light Field Light field: holds all outgoing light rays Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light … Scene modulates outgoing light; light field captures it all.

9. A Big Plenoptic Question: Image entraps a partial scene description –Many-to-One map; 3D->2D –Occlusion hides some scene features –(BRDF * irradiance) tough to split! –limited resolution ? Does Plenoptic fcn. contain full scene ? Two Options for IBMR methods: –Find a limited subset of scene info, –Use MORE than plenoptic function data: (vary lights, etc.) !NO!

10. 8-to-10-Dimensional Ideal? Light field(4D) + light sources(4D) + time + Light field(4D) + light sources(4D) + time + Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light …

11. It gets worse… A ‘Circular problem’: PLUS! depth-of-focus, sampling, indirect illum… Shape BRDF Irradiance Surface Normal

12. Practical IBMR What useful partial solutions are possible? Texture Maps++:Texture Maps++: Image(s)+Depth: (3D shell)Image(s)+Depth: (3D shell) Estimating Depth & SilhouettesEstimating Depth & Silhouettes ‘Light Probe’ measures real-world light‘Light Probe’ measures real-world light Light control measures BRDFLight control measures BRDF Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

13. Texture Maps ++ Re-use rendering results: ‘Impostors’, ‘Billboards’, ‘3D sprites’ Render portion of scene as a textureRender portion of scene as a texture Apply to mesh or plane  to C.O.P.;Apply to mesh or plane  to C.O.P.; Replace if eyepoint changes too muchReplace if eyepoint changes too much

14. Images + Depth 1 Image + Depth: a ‘thin shell’1 Image + Depth: a ‘thin shell’ –Reprojection (well known); Z-buffers can help –McMillan`95: 4-way raster ensures depth order –Problem: ‘holes’, occlusion, matching Multiple Images:Multiple Images: –LDI, LDI trees for multiresolution Limitations:Limitations: –Presumes diffuse-only environment –Depth capture tough: laser TOF reflectometer, manual scanner, structured light, or …

15. Shape Problems: Correspondence Can you find ray intersections? Or ray depth? Ray colors might not match for non-diffuse materials (BRDF)

16. Shape Problems: Correspondence Can you find ray intersections? Or ray depth? Ray colors might not match for non-diffuse materials (BRDF)

17. Estimating Depth, Silhouettes Mildly new IBMR methods can help… Sparse, manual image correspondences (Debevec, Seitz,)Sparse, manual image correspondences (Debevec, Seitz,) Video sequences with camera motion trackingVideo sequences with camera motion tracking Image (silhouette)-based Visual Hulls, ‘voxel carving’ (VIDEO!)Image (silhouette)-based Visual Hulls, ‘voxel carving’ (VIDEO!) Mostly a Classic Computer Vision Problem: Epipolar Geometry: reduce search for correspondencesEpipolar Geometry: reduce search for correspondences Global & local tracking & alignment methods…Global & local tracking & alignment methods…

18. Light Probe: Irradiance Estimate Place mirrored ball in scene,Place mirrored ball in scene, Photograph (careful! High contrast image!)Photograph (careful! High contrast image!) Map position on sphere to incoming angle, intensity to irradiance.Map position on sphere to incoming angle, intensity to irradiance. Repeat where illumination changes greatlyRepeat where illumination changes greatly (in shadows, etc.) Uses: -mixing real & synthetic objects (Ward 96) -separating reflectance & illum (Yu 97) -separating reflectance & illum (Yu 97)

19. Light Control Methods Form estimates of surface properties (BRDF vs. position) by moving camera, light source, or both. Carefully control incoming light direction (light stages, whirling banks of lights, etc)Carefully control incoming light direction (light stages, whirling banks of lights, etc) Establish surface geometry (before, during)Establish surface geometry (before, during) Sort pixels by incoming/outgoing surf. AngleSort pixels by incoming/outgoing surf. Angle Scattered data interpolation to get BRDF.Scattered data interpolation to get BRDF.

20. Conclusion Very active areaVery active area Heavy overlap with computer vision: careful not to re-invent & re-name!Heavy overlap with computer vision: careful not to re-invent & re-name! Compute-intensive, but easily parallel; applies graphics hardware to broader probs.Compute-intensive, but easily parallel; applies graphics hardware to broader probs.