1. 1 Surface Reflectance Estimation and Natural Illumination Statistics Ron Dror, Ted Adelson, Alan Willsky Artificial Intelligence Lab, Lab for Information and Decision Systems http://www.ai.mit.edu/people/rondror July 13, 2001

2. 2 Reflectance Estimation Problem Surface appearance depends on surface reflectance, illumination, and geometry. We wish to estimate reflectance under unknown illumination.

3. 3 Human vision

4. 4 Machine vision

5. 5 Motivation Recognize materials. Reflectance, like texture, is a primary visual characteristic of materials. Material recognition is important in its own right and as a complement to shape recognition. Capture real-world reflectances for rendering purposes. Rectify classical motion, stereo, and shape- from-shading algorithms.

6. 6 Reflectance estimation is ill-posed A surface’s BRDF f(  i,  i ;  r,  r ) specifies how much of the light incident from any one direction is emitted in any second direction. The brightness of a surface patch to a viewer is a weighted integral over illumination from all directions. Goal: estimate reflectance (function of 4 variables) from an image (function of 2 variables) under unknown illumination from every direction (function of 2 variables at every point on the surface). More degrees of freedom than measurements, even assuming known geometry, homogeneous reflectance.

7. 7 Bayesian formulation Find the most likely reflectance given image data. Given image data R, find most likely reflectance f by marginalizing over illumination I. P(f) – prior probability of a reflectance function P(I) – prior probability of an illumination field Challenges: P(f) and P(I) are not readily available. Integration over all illuminations is computationally daunting!

8. 8 Two simplified formulations 1. Classification (finite but arbitrary classes): 2. Parameter estimation using a reflectance model (regression).

9. 9 Prior information: illumination Assuming distant light sources, we can represent illumination by a single spherical image. Projection of spherical map Rendered surfaces

10. 10 Statistical models of illumination Illumination maps possess statistical regularities akin to those of “natural images”. Histogram of pixel intensities Histogram of wavelet coefficients

11. 11 Importance of illumination statistics for humans People recognize reflectance more easily under realistic illumination than simplified illumination.

12. 12 Ward reflectance model A physically realizable variant of the Phong model (satisfies energy conservation and reciprocity).  d : proportion of incident radiation reflected diffusely.  s : proportion of incident radiation reflected specularly.  : surface roughness, or blur in specular component. diffuse component specular lobe

13. 13 Effect of Ward model parameters on pixel intensity histogram Original pixel intensity probability

14. 14 Effect of Ward model parameters on pixel intensity histogram Original  d =.1 pixel intensity probability

15. 15 Effect of Ward model parameters on pixel intensity histogram Original  d =.2 pixel intensity probability

16. 16 Effect of Ward model parameters on pixel intensity histogram Original  d =.3 pixel intensity probability

17. 17 Effect of Ward model parameters on pixel intensity histogram Original  d =.4 pixel intensity probability

18. 18 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.1 pixel intensity probability

19. 19 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.2 pixel intensity probability

20. 20 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.3 pixel intensity probability

21. 21 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.4 pixel intensity probability

22. 22 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.4  =0 pixel intensity probability

23. 23 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.4  =.05 pixel intensity probability

24. 24 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.4  =.1 pixel intensity probability

25. 25 Effect of Ward model parameters on pixel intensity histogram Original  d =.4  s =.4  =.15 pixel intensity probability

26. Dependence of statistics on reflectance parameters Real-world illuminations Random checkerboard illuminations  d  s   d  s  kurt mean skew 10% 50% 90% var kurt mean skew 10% 50% 90% var kurt mean skew 10% 50% 90% var normalized derivative

27. 27 Each reflectance clusters in feature space black matteblack shiny white mattewhite shiny gray shinychrome

28. 28 A system for classification “Learn” relationships between features of the observed image and reflectance classes. For a distant viewer and convex object, radiance depends only on local surface orientation.

29. 29 Implementation flow chart This leaves two open questions: How to select relevant statistics? How to build a classifier?

30. 30 SVM classifier Support vector machines are relatively robust to the inclusion of extraneous features. A sample classifier based on just two statistics: black matteblack shiny white mattewhite shiny gray shinychrome

31. 31 Training data sets 6 Ward model reflectances, 9 illuminations (Debevec) 11 Ward model reflectances, 100 illuminations (Teller) 9 real spheres, photographed at seven locations

32. 32 Performance Rendered: 6 BRDFs, 9 illums Rendered: 11 BRDFs, 100 illums Photos: 9 spheres, 7 illums Chance16.7% 9.1%11.1% 6 hand- selected features 98.1%98.5%93.7% 6 auto- selected features 96.3%94.4%74.6% 6 PCA features 79.6%86.8%71.4%

33. 33 Conclusions Our classifier rivals human performance when geometry is known and reflectance is homogeneous. Although ill-posed, reflectance estimation under unknown natural illumination is tractable. The statistical structure of natural illumination plays an essential role in visual reflectance estimation by humans and machines.

34. 34 Future directions In progress or submitted: Extension to complex or unknown geometry; robustness to incorrect assumed geometry. Quantitative study of natural illumination statistics. Measurement of human ability to estimate reflectance from a single image without contextual information. Additional goals: Rigorous theoretical foundation – link illumination statistics directly to selected features. Estimate spatially varying reflectance.

35. 35 Misclassifications Illumination Misclassified image Potential source of confusion

36. 36 Feature selection By hand, based on insights developed through work with Ward model. Using automated feature selection method, which iterates the following steps: Estimate marginal probability density of each feature for each class. Select the feature that minimizes Bayes error. Regress remaining features against selected features, and subtract off predicted values.

37. 37 Auto-selected features 6 features selected based on images of spheres with 6 Ward model reflectances under 9 illuminations: 10th percentile of 4th finest vertical subband 90th percentile of pixel intensity variance of 3rd finest diagonal subband 10th percentile of pixel intensity 90th percentile of 4th finest vertical subband median of 3rd finest horizontal subband Hand-selected features mean and 10th percentile of original image variance of two finest vertical subbands ratio of these two variances kurtosis of second finest vertical subband

38. 38 Complex vs. simple illumination People recognize reflectance more easily under realistic illumination than simplified illumination. A reflectance estimation algorithm which takes advantage of natural illumination statistics will fail for atypical illumination.

39. 39 Human reflectance estimation Pool balls Note ambiguity in overall color and brightness when matte spheres are viewed in isolation.

40. 40 Related Work Yu, Debevec, Malik, and Hawkins, ’99 Reflectance and illumination from multiple photos. Sato, Wheeler, and Ikeuchi, ’97 Reflectance and geometry from photos and laser range finder, with known illumination. Marschner, Greenberg, et al., ’98, ’99 Reflectance under known illumination. Tominaga and Tanaka, 1999, ’00 Reflectance and geometry under simple lighting, using color separation. Pellacini, Ferwerda, Greenberg, ’00 Perceptually uniform gloss space for graphics. Ramamoorthi and Hanrahan, ’01 Determine when reflectance estimation problem is well- posed.

41. 41 Photographic data Nine different spheres under the same illumination.

42. 42 Photographic data Same spheres under a second illumination.

43. 43 Photographic data Same spheres under a third illumination.

44. 44 Illumination conditions

45. 45 Rendered data set 6 spheres under one illumination condition

46. 46 Rendered data set 6 spheres under a 2nd illumination condition

47. 47 Is this task even possible? Humans are good at it. In psychophysical tests, we found that humans could match synthetic images of surfaces with similar reflectances rendered under different real-world illuminations. Two conclusions: Humans rely on prior information in estimating reflectance. Humans estimate reflectance without explicitly estimating illumination. Photographs of three spheres under two illumination conditions

48. 48 Additional applications Rectify motion, stereo, and shape-from- shading algorithms. Capture real-world reflectances for rendering purposes. Yu, Debevec, Malik, Hawkins, SIGGRAPH 1999 Gideon Stein, unpublished

49. 49 Debevec spheres

50. Dependence of statistics on reflectance parameters Real-world illuminations Random checkerboard illuminations  d  s   d  s  kurt mean skew 10% 50% 90% var kurt mean skew 10% 50% 90% var kurt mean skew 10% 50% 90% var normalized derivative kurt mean skew 10% 50% 90% var kurt mean skew 10% 50% 90% var kurtmeanskew 10% 50% 90% var