Polarization-based Inverse Rendering from Single View Daisuke Miyazaki Robby T. Tan Kenji Hara Katsushi Ikeuchi

2 Modeling cultural assets Integrated framework for obtaining 3 types of information Geometrical Photometrical Environmental

3 Related work GeometryPhotometryEnvironment Tominaga et.al Zheng et.al Nayar et.al Sato et.al Ramamoorthi et.al Nishino et.al Hara et.al Proposed method

4 Outline 1. Reflection components separation 2. Shape from polarization using diffuse light 3. Light source estimation from intensity peak 4. Reflection parameters estimation by l.s.m. Minimize K s, σ rendered image real image 2

1. Reflection components separation

6 Dichromatic reflection model Incident light Specularly reflected light Diffusely reflected light Air Object Surface normal

7 Reflection components separation Diffuse Input Specular [Tan2002] Shape Illumination Reflection parameters

2. Shape from polarization

9 Related work ObjectReflectionView Koshikawa 1979OpaqueSpecular1 Wolff 1990OpaqueDiffuse2 Rahmann et.al. 2001OpaqueDiffuse2~5 Miyazaki et.al. 2002TransparentSpecular2 Proposed methodOpaqueDiffuse1

10 Polarization Incident light Specularly reflected light Diffusely reflected light Air Object

11 Surface normal Object Surface normal Polarizer Camera  Zenith angle  Azimuth angle

12 Azimuth angleφ and intensity difference Rotation angle of polarizer Intensity I max 360  11 22  -ambiguity I min Azimuth angle 

13 Propagation [Ikeuchi&Horn1981] Determination of azimuth angle  Propagate φ from occluding boundary to inner part of object area (Assumption: smooth surface) object Cannot apply to “dimples”(=perfect concave)

14 Zenith angleθ and DOPρ ° Zenith angle θ DOP ρ Degree Of Polarization ρ θ

15 Modification ° Zenith angle θ DOP ρ Degree Of Polarization u: modification factor Raises DOP Assumption Closed smooth object “u” is constant Definition of DOP: Modified DOP: u

16 Surface normal φ θ

17 Height Relaxation method Minimize: where, Gradient Height H Iteratively update: [Ikeuchi1984] Surface normal

3. Illumination estimation

19 Illumination sphere θ=0° θ=90° θ=180° Object Light source is represented in polar coordinate system (θ, φ) φ=0° φ=90° φ=180° φ=270° L 1 =(θ 1, φ 1 ) L 2 =(θ 2, φ 2 ) L 3 =(θ 3, φ 3 )

20 Illumination estimation Detect position of intensity peak Determine light source orientation from the peak 1.Project to (θ, φ)-space 2.Thresholding 3.Detect intensity peak

4. Reflection parameters estimation

22 Torrance-Sparrow reflection model Specular reflectionDiffuse reflection Incident light Surface normal View Bisector Object surface α θiθi θrθr Known: θ i, θ r, α Unknown: Diffuse reflection scale; K d Specular reflection scale; K s Surface roughness; σ

23 Reflection parameters estimation Solve the following least-square problem by steepest-descent method Minimize K s, σ rendered image real image 2

Experimental result

25 Input Intensity I Azimuth angleφ DOPρ

26 Result of shape estimation

27 Result of illumination estimation Actual illumination distribution Estimated illumination distribution

28 Rendering result Input Synthesized image Rendered image under different illumination & view

29 Result for another object Input Synthesized image Estimated shape Rendered image under different illumination & view

30 Conclusions Estimated geometrical, photometrical, environmental information in one integrated framework –Shape from polarization –Surface reflection parameters from iterative computation –Illumination from intensity peak

31 Application to digital archiving project Multiple View Modeling a statue in a room –IBR with surface normal reflection parameters Photorealistic preservation

Fin

(c) Daisuke Miyazaki 2003 All rights reserved. D. Miyazaki, R. T. Tan, K. Hara, K. Ikeuchi, "Polarization-based Inverse Rendering from Single View," in Proceedings of International Symposium on the CREST Digital Archiving Project, pp.51-65, Tokyo, Japan,