1. Computer Graphics Inf4/MSc Computer Graphics Lecture Notes #16 Image-Based Modelling, Rendering and Lighting

2. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #162 Global Illumination and Image-Based Lighting Traditional Computer Graphics involves: –Modelling with matter: geometry with reflectance properties. Image-Based Modelling & Rendering is: –Modelling and rendering with Light, often deriving geometry and materials in the process. Image-Based Lighting allows: –Combination of real and synthetic graphics with consistent illumination, using images as light sources.

3. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #163 Pursuit of Photorealism For complex models, tools can require lot of effort and rendering is very slow. Choice: do usual modelling or just take a photo (that would provide photorealism!) IBMR is about using these photos and transcending their limitations (eg. Light). Don’t need anything like a complete model, but may need a number of photographs…

4. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #164 Pursuit of Photorealism Need enough photographs to give coverage. Major result is - rendering is faster. So Image-Based Modelling and Rendering (IBMR) starts with the answer! Research inspired by film industry. Related to techniques in Computer Vision. New so no taxonomy, just techniques.

5. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #165 IBMR Techniques Panoramas – movement within and between panoramic cylinders, eg. Quicktime VR. –Panoramas can be real or synthetic. View interpolation –from a few photos, interpolate view from any position (morphing). * Lighting – adding real or synthetic objects to a photographed scene, illuminated with scene lights. Modelling – extracting models from photos.

6. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #166 “Traditional graphics”

7. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #167 Computer Vision

8. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #168 Vision meets graphics

9. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #169 However... Vision falls short

10. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1610 And... … so does graphics

11. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1611 Image-Based Rendering (IBR)

12. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1612 What is an image? Collection of radiance values radiance along a ray 3D position 2D direction

13. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1613 Plenoptic function Radiance value for all possible rays = plenoptic function. All possible images are a subset of this plenoptic function. Too much stuff! Goal of IBR is generate a continuous representation of the plenoptic function.

14. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1614 Plenoptic function Radiance is constant along a ray (line) 2D position 2D direction Once we know one ‘origin’, we know them all

15. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1615 What is an image? Image = rays going through one point usually restricted to viewing frustum, but can also be panoramic

16. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1616 What is an object? Image = rays going through one point + image plane 2D function (position on image plane)

17. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1617 What is an object? Outgoing radiance field of an object 2D function (position on surface)

18. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1618 What is an object? All light leaving the object

19. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1619 What is an object? All light leaving the object 4D function (2D position + 2D direction)

20. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1620 What is an object? All possible images of an object

21. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1621 What is an object? All possible images of an object

22. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1622 What is an object? We don’t really need the object

23. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1623 What is an object? We don’t really need the object

24. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1624 Lumigraph / Light Field Object is only defined by its radiance field stuff 4D function (Levoy - Cohen et al 96)

25. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1625 Lumigraph - capture (Stanford - Levoy et al.)

26. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1626 Layered Depth Images (McMillan) Problem with one photo is “holes” when view point moved: - this is why a number of photos are generally required - fill the holes using intensities of neighbouring pixels - interpolation …

27. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1627 Image Based Lighting Add models or objects to scenes and allow them to be manipulated in the scenes. Modelling with light allows added objects to be illuminated consistent with image existent lighting - photorealistic inclusions. We start with the answer by finding the scene illumination.

28. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1628 Image Based Lighting Real Scene Goal: place synthetic objects on table (all pictures P. Debevec 98-99)

29. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1629 Extracting scene lighting Capture illumination using illumination sphere

30. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1630 Image Based Lighting Real scene

31. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1631 Image Based Lighting captured illumination field

32. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1632 Image Based Lighting Real scene light based model local scene synthetic objects

33. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1633 Image Based Lighting Use renderer - compute effects of synthetic objects on local scene local scene (brdf estimated)synthetic objects (brdf known) light based model

34. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1634 Image Based Lighting Render into the scene background

35. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1635 Image Based Lighting Render synthetic objects

36. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1636 Image Based Lighting Effect of local scene on real scene

37. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1637 Image Based Lighting Add differences to image

38. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1638 Reconstruction from images? Computer Vision related. Constructs new views and extracts models. Epipolar geometry –expresses relationships between points in different images. Difficult to predict full impact in CG. State of the Art –3D reconstruction from uncalibrated images.

39. Computer Graphics Inf4/MSc 13/11/2007Lecture Notes #1639 References You won’t find this material in the major graphics texts since it is new and not yet mainstream but I recommend the following: –The Computer Image, Watt & Policarpo, Addison- Wesley 1998. –SIGGRAPH courses for the past few years, we have some on CD rom and on-line within EdVEC and not immediately available to Informatics machines – ask me if you have a strong interest and need.