1. Inferring Reflectance Functions from Wavelet Noise Pieter Peers Philip Dutré Pieter Peers Philip Dutré June 30 th 2005 Department of Computer Science

2. Image-based Relighting / Environment Matting Scene (fixed viewpoint)

3. Image-based Relighting / Environment Matting … Scene (fixed viewpoint) Novel Incident Illumination +

4. Image-based Relighting / Environment Matting … … Scene (fixed viewpoint) Novel Incident Illumination Compute Relit Image + =

5. Image-based Relighting / Environment Matting … … Scene (fixed viewpoint) Novel Incident Illumination Compute Relit Image + = Reflectance Function

6. Examples of Reflectance Functions Diffuse Ball Specular Ball

7. Examples of Reflectance Functions Diffuse Ball Specular Ball

8. Examples of Reflectance Functions Diffuse Ball Specular Ball Reflectance Function

9. Reflectance Functions (frequency domain) Diffuse Ball Specular Ball Reflectance Function (frequency domain)

10. Reflectance Functions (wavelet domain) Diffuse Ball Specular Ball Reflectance Function (wavelet domain)

11. Relight a Pixel Novel Incident Illumination Specular Ball Relit pixel value? Reflectance Function (wavelet space)

12. Relight a Pixel Novel Incident Illumination Specular Ball Reflectance Function (wavelet space)Incident Illumination (wavelet space)

13. Relight a Pixel Novel Incident Illumination Specular Ball Reflectance Function (wavelet space)Incident Illumination (wavelet space) ((  )

14. Relight a Pixel Novel Incident Illumination Specular Ball Reflectance Function (wavelet space)Incident Illumination (wavelet space) ((  ) Only non-zero coefficients

15. Directly Observing Reflectance Functions Controlled Incident Illumination Photograph of Specular Ball Emit (e.g. from CRT)

16. Directly Observing Reflectance Functions Controlled Incident Illumination Photograph of Specular Ball Reflectance Function (unknown) Observed pixel Controlled Incident Illumination (wavelet space)

17. Directly Observing Reflectance Functions Controlled Incident Illumination Photograph of Specular Ball Unknown Reflectance Function (wavelet space) ((  ) Controlled Incident Illumination (wavelet space)

18. Directly Observing Reflectance Functions Controlled Incident Illumination Photograph of Specular Ball Controlled Incident Illumination (wavelet space) ((  ) Only non-zero coefficients Observed coefficient Unknown Reflectance Function (wavelet space)

19. Number of Observations Specular Ball Reflectance Function (wavelet space) #Photographs = #Illumination pixels Incident Illumination

20. Number of Observations Problem Specular Ball Reflectance Function (wavelet space) Incident Illumination 1000 x 1000 #Photographs = #Illumination pixels

21. Wavelet Noise Illumination Wavelet Noise Normal distribution of wavelet coefficients Mean : 0.0 Standard deviation : 1.0 Rescale Wavelet Noise Pattern to fit into [0..1] range Wavelet Noise Pattern Wavelet Noise Pattern (wavelet space) Advantages Arbitrary number of different patterns possible Any reflectance function gives a non-zero response Constant average luminance

22. Estimating Wavelet Coefficients (Unknown) Reflectance Function Wavelet Noise Assume: positions of are known Question: what are the magnitudes? ((  ) = Observed Pixel Value

23. Estimating Wavelet Coefficients ( )  = Leave out zero coefficients (of the reflectance function) Wavelet Noise (linearized) Reflectance Function (linearized) Observed Pixel Value

24. Estimating Wavelet Coefficients  = … Multiple observations matrix-vector multiplication … Wavelet Noise Reflectance Function Observed Pixel Values # emitted patterns # observations

25. Estimating Wavelet Coefficients  = Finding magnitudes : Linear Least Squares problem … … Wavelet Noise Reflectance Function Observed Pixel Values

26. Estimating Wavelet Coefficients  = Estimation error when only a part is approximated? … … Wavelet Noise Reflectance Function Observed Pixel Values

27. Partial Estimation  + … …… = = … Wavelet Noise Reflectance Function Observed Pixel Values

28. Partial Estimation According to a normal distribution  + … …… = = … Wavelet Noise Reflectance Function Observed Pixel Values

29. Partial Estimation According to a normal distribution  + … …… = = … Wavelet Noise Reflectance Function Observed Pixel Values Normal distribution

30. Partial Estimation  + … … = = … Wavelet Noise Reflectance Function Observed Pixel Values Finding the best approximation for : Linear Least Squares problem NoIseNoIse

31. Inferring Reflectance Functions Reflectance Function (2D wavelet space) Priority Queue of Candidates

32. Inferring Reflectance Functions Reflectance Function (2D wavelet space) Priority Queue of Candidates

33. Inferring Reflectance Functions Reflectance Function (2D wavelet space) Priority Queue of Candidates

34. Reflectance Function (2D wavelet space) Inferring Reflectance Functions Priority Queue of Candidates

35. Inferring Reflectance Functions Reflectance Function (2D wavelet space) Priority Queue of Candidates

36. Inferring Reflectance Functions Reflectance Function (2D wavelet space) Priority Queue of Candidates

37. Inferring Reflectance Functions Reflectance Function (2D wavelet space) Priority Queue of Candidates

38. Inferring Reflectance Functions Reflectance Function (2D wavelet space) Priority Queue of Candidates

39. Overview Record photographs Emit Wavelet Noise Predetermined number of photographs

40. Overview Record photographs Infer Reflectance Functions Reflectance Function Progressive Algorithm For each pixel

41. Overview Record photographs Infer Reflectance Functions Compute Relit Image Relight Incident Illumination

42. Results 64 Haar Wavelet Coefficients 256 Photographs Reference Photograph

43. Results 64 Haar Wavelet Coefficients 256 Photographs Reference Photograph

44. Results 64 Haar Wavelet Coefficients 256 Photographs Reference Photograph

45. Results 64 Haar Wavelet Coefficients 256 Photographs Reference Photograph

46. Results 64 Haar Wavelet Coefficients 256 Photographs Reference Photograph

47. Results 128 Haar Wavelet Coefficients 512 Photographs Reference Photograph

48. Results: High Frequency Illumination

49. Conclusion & Future Work Inferring Reflectance Functions from Wavelet Noise –No restriction on material properties –Stochastic illumination patterns –Trade-off quality versus acquisition time Future Work –Noise filtering –Higher-order wavelets