1. Interreflections : The Inverse Problem Lecture #12 Thanks to Shree Nayar, Seitz et al, Levoy et al, David Kriegman

2. Graphics

3. Vision: Estimating Shape of Concave Surfaces Actual Shape Shape from Photometric Stereo Need to account for Interreflections!!

4. Shape and Interreflections: Chicken and Egg If we remove the effects of interreflections, we know how to compute shape. But, interreflections depend on the shape!! So, which comes first?

5. Linear System of Radiosity Equations - RECAP Known Unknown Matrix Inversion to Solve for Radiosities.

6. Use Radiance instead of Radiosity Vision shape-from-intensity algorithms work on Radiance. Assume image pixel covers infinitesimal scene patch area. Form factor is called “Interreflection Kernel”, K (better name). Radiance of a facet is given by the linear combination of radiances from other facets. Loosely, we can say, this weighted averaging in the direction of concave curvature.

7. Why do concavities appear shallow? Radiance of a facet is given by the linear combination of radiances from other facets. Loosely, we can say this weighted averaging in the direction of concave curvature.

8. Matrix Form of Interreflection Equation

9. Pseudo Shape and Reflectance Apply any shape from intensity algorithm ignoring interreflections! Actual Shape Pseudo-shape from Photometric Stereo KEY IDEA: The pseudo shape and reflectance (albedo) is related to the actual shape and reflectance.

10. Pseudo Shape/Reflectance from Photometric Stereo Facet Matrix : Source direction Three Source Directions :

11. Pseudo Shape/Reflectance from Photometric Stereo Three Source Directions : Pseudo Shape :

12. Key Observations Pseudo Facets: Lambertian! “Smoothed” versions of actual facets (shallow) Pseudo albedos may be greater than 1. Pseudo Shape and Albedos are independent of source direction! This allows us to reconstruct actual shape.

13. Iterative Refinement of Shape and Albedos Start with pseudo shape and albedos as initial guesses. Compute Interreflection Kernel K, and Albedo matrix P. Iterate until convergence.

15. Important Assumptions and Observations Any shape-from-intensity method can be used. Assumes shape is continuous (for integrability). All facets contributing to interreflections must be visible to sensor. Facets are infinitesimal lambertian patches. Complexity: Convergence shown for 2 facets. Does not always converge to the right facet for large tilt angles (> 70). (M Iterations, n facets)

19. 3D Objects with Translational Symmetry

20. 3D Objects with Translational Symmetry

23. Shape and Interreflections: Chicken and Egg If we remove the effects of interreflections, we know how to compute shape. But, interreflections depend on the shape!! So, which comes first?

24. Removing Interreflection Images by Ward et al., SIGGRAPH 88

25. Bounce Images

26. Main Results There exists a matrix C 1 that removes all interreflections in a photograph (or lightfield) Works for any illumination There is a matrix C k that retains only the k th bounce

27. Light transport T

28. The transport matrix = TL in L out Accounts for interreflections, shadows, refraction, subsurface scatter,... [Dorsey 94] [Zongker 99] [Debevec 00] [Peers 03] [Goesele 05] [Sen 05]...

29. The transport matrix = TL in L out

30. Direct illumination rendering = T1T1 L in L 1 out Single bounce from light to eye no interreflections BRDF

31. Inverse rendering = T -1 L in L out

32. Removing Interreflections L 1 out L out = T -1 T1T1 C1C1 Second derivation: from the rendering equation [Kajiya 86] [Cohen 86]

33. Cancellation Operators Recursively define other operators ( I – C 1 ) –gives interreflected light C 2 = C 1 ( I – C 1 ) –gives second bounce of light C k = C 1 ( I – C 1 ) k-1 –gives k th bounce of light Inverse ray tracing!

34. How to compute C 1 Simplified case Lambertian reflectance and fixed viewpoint L in and L out are 2D Can capture T by scanning a laser

35. Synthetic M Scene TC4TC4TC3TC3TC2TC2TC1TC1T 1 23 4 T (log-scaled) 1 2 3 4

36. TC4TC4TC3TC3TC2TC2TC1TC1T real data

38. (x3) (x6)(x15) direct indirect

39. direct indirect (x3) (x6)(x15) flash light illumination

40. Shape and Interreflections: Chicken and Egg If we remove the effects of interreflections, we know how to compute shape. But, interreflections depend on the shape!! So, which comes first?

41. Dual Photography Pradeep Sen, Billy Chen, Gaurav Garg, Steve Marschner, Mark Horowitz, Marc Levoy, Hendrik Lensch

42.  Marc Levoy Related imaging methods time-of-flight scanner –if they return reflectance as well as range –but their light source and sensor are typically coaxial scanning electron microscope Velcro® at 35x magnification, Museum of Science, Boston

43. The 4D transport matrix scene photocell projector camera

44. The 4D transport matrix scene projector P pq x 1 C mn x 1 T mn x pq

45. T PC = The 4D transport matrix pq x 1 mn x 1 mn x pq

46. T C = The 4D transport matrix 1000010000 mn x pq pq x 1 mn x 1

47. T C = The 4D transport matrix 0100001000 mn x pq pq x 1 mn x 1

48. T C = The 4D transport matrix 0010000100 mn x pq pq x 1 mn x 1

49. The 4D transport matrix T P C = pq x 1 mn x 1 mn x pq

50. The 4D transport matrix applying Helmholtz reciprocity... T PC = pq x 1 mn x 1 mn x pq T P’C’ = mn x 1 pq x 1 pq x mn T

51.  Marc Levoy Example conventional photograph with light coming from right dual photograph as seen from projector’s position

52.  Marc Levoy Properties of the transport matrix little interreflection → sparse matrix many interreflections → dense matrix convex object → diagonal matrix concave object → full matrix Can we create a dual photograph entirely from diffuse reflections?

53.  Marc Levoy Dual photography from diffuse reflections the camera’s view

54.  Marc Levoy The relighting problem subject captured under multiple lights one light at a time, so subject must hold still point lights are used, so can’t relight with cast shadows Paul Debevec’s Light Stage 3

55. The 6D transport matrix

57.  Marc Levoy The advantage of dual photography capture of a scene as illuminated by different lights cannot be parallelized capture of a scene as viewed by different cameras can be parallelized

58. scene Measuring the 6D transport matrix projector camera arraymirror arraycamera

59.  Marc Levoy Relighting with complex illumination step 1: measure 6D transport matrix T step 2: capture a 4D light field step 3: relight scene using captured light field scene camera array projector T P’ C’ = mn x uv x 1pq x 1 pq x mn x uv

60.  Marc Levoy Running time the different rays within a projector can in fact be parallelized to some extent this parallelism can be discovered using a coarse-to-fine adaptive scan can measure a 6D transport matrix in 5 minutes

61. Can we measure an 8D transport matrix? scene camera arrayprojector array