1. Polarization-based Shape Estimation of Transparent Objects by Using Raytracing and PLZT Camera
Daisuke Miyazaki The University of Tokyo
Noriyuki Takashima Furuuchi Chemical Corporation
Akira Yoshida Furuuchi Chemical Corporation
Eiki Harashima Furuuchi Chemical Corporation
Katsushi Ikeuchi The University of Tokyo
I'm Daisuke Miyazaki. and this is a joint work with noriyuki takashima, akira yoshida, eiki harashima, and katsushi ikeuchi.

2. The 1st part of this talk PLZT polarization camera
Introduction(1/2)
PLZT polarization camera(12)
Inverse polarization raytracing(11)
Conclusion(2)
The 1st part of this talk
PLZT polarization camera
Measures the polarization state (Stokes vector) of the light
Is controllable from the computer
The talk consists of two parts. for the 1st part, i will talk about our plzt polarization camera. this camera measures the polarization state of the observed light. the polarization state of the light is expressed in stokes vector. this camera is electrically controllable from the computer. the camera measures the stokes vector semi-automatically. we use a material called plzt for the filter of the camera.

3. The 2nd part of this talk Inverse polarization raytracing
Introduction(2/2)
PLZT polarization camera(12)
Inverse polarization raytracing(11)
Conclusion(2)
The 2nd part of this talk
Inverse polarization raytracing
Estimates the 3D shape of transparent objects
Solves the inverse problem of the polarization raytracing
the 2nd part of this talk is about inverse polarization raytracing. the purpose is to estimate the 3d geometrical shape of transparent objects; for example, objects made of glass, acrylic, and so on. since the target object is transparent, we must compute the direction of the light and the polarization state of the light. we use the polarization raytracing to calculate the polarization state of the light ray. here we use a simple combination of raytracing algorithm and mueller calculus. we estimate the shape by minimizing the difference between the rendered data and the measured data.

4. PLZT polarization camera
The first part of this talk is about plzt polarization camera. this camera measures the stokes vector of the observed light.

5. PLZT polarization camera(1/12)
Introduction(2)
PLZT polarization camera(1/12)
Inverse polarization raytracing(11)
Conclusion(2)
Mueller calculus
Intro(1/2)
Algorithm(6)
Experiment(4)
Light: 4D vector (Stokes vector)
Material: 4x4 matrix (Mueller matrix)
Intensity
Power of 0 linear polarized light
Power of 45 linear polarized light
Power of right circular polarized light
DOP
(degree of polarization)
first of all, i would like to briefly overview the mueller calculus. mueller calculus is a method to calculate the polarization state of the light. the light is represented in 4d vector. this vector is called stokes vector. the material which changes the state of the light is represented in a 4 by 4 matrix. this matrix is called mueller matrix. here is the definition of the stokes vector. stokes vector is consisted of 4 components. the first component s0 represents the intensity of the light. the second component s1 represents the strength of the energy of horizontal linear polarized light. the third component s2 represents the strength of the energy of 45 degree diagonal linear polarized light. the fourth component s3 represents the strength of the energy of right circular polarized light. DOP is short of degree of polarization, and is calculated from stokes vector as shown here.

6. PLZT polarization camera(2/12)
Introduction(2)
PLZT polarization camera(2/12)
Inverse polarization raytracing(11)
Conclusion(2)
Mueller matrix
Intro(2/2)
Algorithm(6)
Experiment(4)
ND (neutral density) filter
WN: alpha value (0~1)
Horizontal linear polarizer
WL: 0~0.5 (ideally 0.5)
Retardation
δ: retardation value
Rotation
here is some examples of mueller matrices. nd filter, neutral density filter, is a black translucent filter to reduce the brightness of the light. the mueller matrix of nd filter is expressed with large n. this matrix is an identity matrix scaled by the value wn. wn varies from 0 to 1, and represents the translucency of the nd filter. the value wn is determined a priori for the actual nd filter we use. mueller matrix of linear polarizer whose fast axis is the same as x axis is expressed with large l. here, wl is a value which varies from 0 to 0.5. for an ideal linear polarizer, wl would be 0.5. the value wl is determined a priori for the actual linear polarizer we use. the mueller matrix of retarder is expressed with large d. retardation plate chages the phase of the light wave. delta represents the amount of the phase shift. this mueller matrix represents the rotation, and is expressed with large c. the rotation angle is represented as phi. this rotation matrix is used for changing the coordinates system of the light.

7. PLZT polarization camera(3/12)
Introduction(2)
PLZT polarization camera(3/12)
Inverse polarization raytracing(11)
Conclusion(2)
PLZT
Intro(2)
Algorithm(1/6)
Experiment(4)
Lanthanum-modified lead zirconate titanate
Made from 4 kinds of metal compound Pb: lead La: lanthanum Zr: zirconium Ti: titanium
Transparent ceramics
Birefringent media depending on the voltage
here, i will explain the material called plzt. the name, plzt, is an alias of lanthanum-modified lead zirconate titanate. plzt is made from 4 kinds of metal compound, lead, lanthanum, zirconium, and titanium. plzt is the capital letters of these four materials. plzt is a transparent ceramics. plzt changes its material state depending on the voltage. plzt has a birefringency, and behaves as a retarder. plzt changes the phase of the transmitting light. the amount of the retardation depends on the voltage of the plzt.

8. PLZT polarization camera(4/12)
Introduction(2)
PLZT polarization camera(4/12)
Inverse polarization raytracing(11)
Conclusion(2)
PLZT and ND filter
Intro(2)
Algorithm(2/6)
Experiment(4)
Camera
Target scene
(Light, object, ...)
PLZT ND
filter x y z
+90
Optical
axis
first, we set plzt and nd filter in front of the camera. the light first goes through the plzt, next, goes through the nd filter, and finally, reaches at the camera. the coordinates system is defined as shown here. z axis is heading towards the camera. x axis is the horizontal direction of the camera. the optical axis of the plzt is set orthogonal to x axis. it means that the plzt is rotated 90 degrees from x axis.

9. Mueller matrix of the system
Introduction(2)
PLZT polarization camera(5/12)
Inverse polarization raytracing(11)
Conclusion(2)
Mueller matrix of the system
Intro(2)
Algorithm(3/6)
Experiment(4)
Retardation
Rotation
ND filter
PLZT rotated 90
d: Amount of the phase shift of PLZT
(depends on the voltage)
System
plzt acts as a retardation plate, so the mueller matrix of plzt is expressed as large d. but the plzt is rotated 90 degrees, so we rotate the mueller matrix by rotation matrix, large c. the mueller matrix of the plzt rotated 90 degrees is expressed as d hat. mueller matrix of nd filter is expressed as large n. the light first goes through the plzt and next goes through the nd filter. so the mueller matrix of the whole system can be calculated by multiplying n and d hat. the mueller matrix of the system is expressed as m nd. plzt acts as a retarder and the amount of the retardation is expressed as delta. the value delta depends on the voltage of plzt.

10. PLZT and linear polarizer
Introduction(2)
PLZT polarization camera(6/12)
Inverse polarization raytracing(11)
Conclusion(2)
PLZT and linear polarizer
Intro(2)
Algorithm(4/6)
Experiment(4)
Camera
Target scene
(Light, object, ...)
PLZT
Linear
polarizer
+22.5 x y z
+90
Optical
axis
next, we set plzt and linear polarizer in front of the camera. here, the linear polarizer is used instead of the nd filter. the light first goes through the plzt, next, goes through the linear polarizer, and finally, reaches at the camera. the coordinates system is defined as shown here, which is the same coordinates as the previous slide. the optical axis of the plzt is rotated 90 degrees, same as the previous slide. the optical axis of the linear polarizer is rotated 22.5 degrees from x axis.

11. Mueller matrix of the system
Introduction(2)
PLZT polarization camera(7/12)
Inverse polarization raytracing(11)
Conclusion(2)
Mueller matrix of the system
Intro(2)
Algorithm(5/6)
Experiment(4)
Horizontal linear polarizer
Rotation
Linear polarizer rotated 22.5
PLZT rotated 90
d: Amount of the phase shift of PLZT
(depends on the voltage)
System
the mueller matrix of linear polarizer, whose optical axis is the same as x axis, is expressed as large l. but the linear polarizer in this system is rotated 22.5 degrees, so we rotate the mueller matrix by rotation matrix, large c. the mueller matrix of the linear polarizer rotated 22.5 degrees is expressed as l hat. mueller matrix of plzt rotated 90 degrees is expressed as d hat. the light first goes through the plzt and next goes through the linear polarizer. so the mueller matrix of the whole system can be calculated by multiplying l hat and d hat. the mueller matrix of the system is expressed as m pl. plzt acts as a retarder and the amount of the retardation is expressed as delta. the value delta depends on the voltage of plzt. so, by changing the voltage, plzt acts like a quarter waveplate, half waveplate, and so on.

12. Computing Stokes vector
Introduction(2)
PLZT polarization camera(8/12)
Inverse polarization raytracing(11)
Conclusion(2)
Computing Stokes vector
Intro(2)
Algorithm(6/6)
Experiment(4)
PLZT with ND filter
PLZT with linear polarizer [d retarder]
PLZT with linear polarizer [1/4 waveplate]
PLZT with linear polarizer [1/2 waveplate]
Stokes vector
from 4 images
Inverse
matrix
here is the method to obtain the stokes vector of observed light. the stokes vector of the observed light is expressed as s0, s1, s2, s3. first we observe the light with plzt and nd filter. the mueller matrix of this case is m nd. the stokes vector of the light which comes into the camera can be calculated by multiplying this mueller matrix and the stokes vector of the incoming light. the camera can only observe the intensity of the light, it means that the camera can only observe the first component of the stokes vector, s0, because only s0 represents the intensity of the light. the intensity observed by the camera in this case is expressed as s0,nd. next, we observe the light with plzt and linear polarizer. here we don't set any voltage to the plzt. in this case, the plzt acts as a retarder, and the amount of the retardation is expressed as delta. the value delta is determined a priori. the mueller matrix of this case is m pl delta. the intensity observed by the camera in this case is expressed as s0,pl,delta. then, we change the voltage of the plzt so that the plzt acts like a quarter waveplate. the amount of the voltage is determined a priori. the mueller matrix of this case is m pl half-pi. the intensity observed by the camera in this case is expressed as s0,pl.quarter. finally, we change the voltage of the plzt so that the plzt acts like a half waveplate. the amount of the voltage is determined a priori. the mueller matrix of this case is m pl pi. the intensity observed by the camera in this case is expressed as s0,pl,half. so, we obtain four equations with four unknown parameters; the four components of the stokes vector. by concatenating these four equations, we can express the problem in a matrix equation. by calculating the inverse of this matrix, we can compute the stokes vector of the observed light. that is the algorithm to measure the stokes vector of the observed light.

13. PLZT polarization camera(9/12)
Introduction(2)
PLZT polarization camera(9/12)
Inverse polarization raytracing(11)
Conclusion(2)
Experiment setup
Intro(2)
Algorithm(6)
Experiment(1/4)
Camera
Slider
ND filter
Linear polarizer
PLZT unit
Band-pass filter
UV-cut filter
IR-cut filter +x -x +y -y
here is the measurement system. there is a slider in front of the camera, and the slider can switch two filters, nd filter and linear polarizer. plzt is in front of the slider. there is some filters in front of the plzt, in order to observe only the light with appropriate wavelength. in this system, the light whose wavelength is around 550nm is observed.

14. PLZT polarization camera(10/12)
Introduction(2)
PLZT polarization camera(10/12)
Inverse polarization raytracing(11)
Conclusion(2)
Experiment result
Intro(2)
Algorithm(6)
Experiment(2/4) s0 s1 s2 s3
DOP
here is the measurement result. the target scene is an indoor scene with computer and liquid crystal monitor. the images of the computed stokes vector are shown in this slide. the image of s0 is like this, s1 is this, s2 is this, and s3 is this. the degree of polarization is shown in this image. the liquid crystal monitor is a linearly polarized light, so, the degree of polarization is higher than other area.

15. PLZT polarization camera(11/12)
Introduction(2)
PLZT polarization camera(11/12)
Inverse polarization raytracing(11)
Conclusion(2)
Evaluation
Intro(2)
Algorithm(6)
Experiment(3/4)
DOP of linear polarizer
True value = 1.0
Measurement result = 0.72
s3/s0 of left circular polarizer
True value = -1.0
Measurement result = -0.25
here is the detail of another experiments. first we measured the linearly polarized light and calculated the degree of polarization. the result was 0.72 where the true value was 1. next we measured the left circular polarized light and calculated the fourth parameter of normalized stokes vector. the result was where the true value was -1. unfortunately, the accuracy of the system is not so good especially for the circular polarized light. to improve the accuracy of the system is our future work.

16. PLZT polarization camera(12/12)
Introduction(2)
PLZT polarization camera(12/12)
Inverse polarization raytracing(11)
Conclusion(2)
Related work
Intro(2)
Algorithm(6)
Experiment(4/4)
Liquid crystal polarization camera
Wolff, Mancini, Pouliqen, Andreou (1997)
Fujikake, Takizawa, Aida, Kikuchi, Fujii, Kawakita (1998)
Harnett, Craighead (2002)
Our PLZT polarization camera
Obtain whole Stokes parameters
PLZT has higher response time than LC
some researchers developed a liquid crystal polarization camera. we used plzt instead of liquid crystal. we can obtain all four parameters of stokes vector by using plzt, while they can obtain only the first three parameters of stokes vector by using liquid crystal. another advantage of plzt is that the response is faster than liquid crystal. so, there is a potential that the system with plzt measures the stokes vector of the light faster than the system with liquid crystal.

17. Inverse polarization raytracing
that's the end of the first topic of my presentation. the next topic of my presentation is about the inverse polarization raytracing. here, i propose a method to estimate the 3d geometrical shape of transparent objects.

18. Reflection and transmission
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(1/11)
Conclusion(2)
Reflection and transmission
Intro(1/4)
Algorithm(3)
Experiment(4)
Normal
Depends
upon
Light
Partially polarized
Unpolarized
Air
Object
here is the illustration of the air and the transparent object. (click) suppose that we illuminate the object with unpolarized light. (click) because the object is transparent, one part of the light reflects and the other transmits. both of the light will be partially polarized. (click) how the light polarizes depends on the surface normal. it means that by analyzing the polarization state of the reflected and transmitted light, we can estimate the surface normal, that is, we can estimate the shape of the object.
Partially polarized

19. Inverse polarization raytracing(2/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(2/11)
Conclusion(2)
Interreflection
Intro(2/4)
Algorithm(3)
Experiment(4)
both reflection and transmission occur in transparent object. the method proposed last year only considered the reflection. so the last year's method couldn't estimate the shape of the transparent object in enough precision. in this talk, i will present our new method which considers both reflection and transmission. considering the multiple interreflection inside transparent object is done by raytracing algorithm. the raytracing algorithm is famous in computer graphics field, and calculates the position and the direction of the light ray.
[Miyazaki 2004]
Reflection only
[This method]
Reflection & transmission

20. Polarization raytracing
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(3/11)
Conclusion(2)
Polarization raytracing
Intro(3/4)
Algorithm(3)
Experiment(4)
Raytracing with polarization
Gondek et al. 1994, Wolff & Kurlander 1990, Tannenbaum et al. 1994, Guy & Soler 2004, Chipman 1995, Wilkie 2001
Commercial software
We use: raytracing + Mueller calculus
But we must also calculate the polarization state of the light in addition to the path of the light. Some researchers and software used the method which both calculates the polarization state of the light and the path of the light. in this research, we use a simple combination of raytracing algorithm and mueller calculus.

21. Reflection/Transmission matrix
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(4/11)
Conclusion(2)
Reflection/Transmission matrix
Intro(4/4)
Algorithm(3)
Experiment(4)
Reflection
Transmission
the mueller matrices of reflection and transmission are shown in this slide. here, r and t is derived by the fresnel law. i won't explain the detail of these values in this talk.
Fresnel coefficients:

22. Inverse polarization raytracing(5/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(5/11)
Conclusion(2)
Cost function
Intro(4)
Algorithm(1/3)
Experiment(4)
Input
Calculated
Relationship
between
normal & height
min
dxdy
here is the cost function which we want to minimize in order to estimate the 3d shape of transparent object. (click) the first term of the cost function is the difference between the input polarization data and the calculated polarization data. the input polarization data is obtained by the measurement system, which i will explain later, and the calculated polarization data is obtained by the polarization raytracing method. (click) the second term of the cost function is the relationship between the surface normal and the height. the shape of the object is represented as a height value set for each pixels. the height is expressed as h. the partial differentiation of h by x and y are called the gradient, and they are expressed as p and q. the gradients p and q represent the surface normal. (click) so, the cost function is expressed in this formula. by minimizing this cost function, we can obtain the height and the surface normal of the object, which means that we can obtain the 3d geometrical shape of the transparent object.
Calculate height and normal

23. Inverse polarization raytracing(6/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(6/11)
Conclusion(2)
Update normal
Intro(4)
Algorithm(2/3)
Experiment(4)
Input DOP
Calculated DOP
Light ray
Change
normal
Error
Object
Ray changes
Ray changes
The detail of the algorithm to estimate the surface normal is presented in this slide. the camera observes the light coming from the object. both the reflected light and the transmitted light are observed by the camera. (click) if we virtually change the surface normal of the object surface, the direction of the reflected light and the transmitted light also changes. (click) so, the degree of polarization calculated by the polarization raytracing method changes. by comparing the input value and the computed value, we can find the better surface normal for this surface point. (click) the error can be calculated as a difference between the input value and the computed value. the problem is to find the surface normal which minimizes the error. the problem can be easily solved by the numerical method. that is the algorithm to modify the surface normal.

24. Inverse polarization raytracing(7/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(7/11)
Conclusion(2)
Algorithm overview
Intro(4)
Algorithm(3/3)
Experiment(4)
Initial height
Normal from
height
Minimize
Update normal 2
Input
Calc.
Output height
is small 2
Input
Calc.
Stop when
Height from
normal
the whole algorithm is shown here. (click) first, we set an initial value for the height of the object. (click) second, we calculate the surface normal by differentiating the height value. (click) next, we update the surface normal by minimizing the difference between the input data and the computed data. (click) then, we calculate the height by integrating the surface normal. (click) After that, we check the value of the difference of input data and the computed data. (click) if the error is not enough small, we again update the surface normal. this iteration continues until the error becomes enough small. (click) after the convergence, we finally obtain the 3d geometrical shape of the transparent object.

25. Inverse polarization raytracing(8/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(8/11)
Conclusion(2)
Experimental setup
Intro(4)
Algorithm(3)
Experiment(1/4)
Monochrome
camera
IR/UV cut-off
filter
Linear
polarizer
Transparent object inside
40W
lamp
Plastic
sphere
This is the measurement system to obtain the input polarization data. the target transparent object is set inside this white plastic sphere. this white plastic sphere is illuminated by 36 incandescent lamps. the plastic sphere diffuses the light, and acts like a spherical light source. this provides a uniform illumination distribution. there is a hole on top of the sphere, and the object is observed from the upper direction by the camera and the polarizer. in this experiment, we use a linear polarizer instead of the plzt polarization camera. here, we assume that the target object does not cause a circular polarized light.

26. Inverse polarization raytracing(9/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(9/11)
Conclusion(2)
Experimental result
Intro(4)
Algorithm(3)
Experiment(2/4)
Acrylic hemisphere (r=15mm)
10 loop
Initial (Miyazaki 2004)
50 loop
Error(height)
2.8mm mm
Error(normal)
14 
Frontal shape(estimated)
Frontal shape(truth)
Rear shape(known)
Refractive index 1.5
& Illumination (known)
Here is the measurement result. here, we measured an acrylic hemisphere whose diameter is 30mm. the refractive index was 1.5. we assumed that the refractive index and the illumination distribution are given. this is the estimated shape of this object. the iteration is done 10 times for this result. the cross section of this result is shown here. the blue line is the shape of the bottom surface, which we assumed to be given. the red line is the true shape of the top surface. the white line is the estimated shape of the top surface. the left picture shows the initial shape. here, we used the result of the previous method as an initial shape. the right picture is the result after 50 iteration. see that the estimated shape, white line, is similar to the true shape, red line. the rms error of height and surface normal is shown here. see that the error of the proposed method are decreased from the result of the previous method.

27. Inverse polarization raytracing(10/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(10/11)
Conclusion(2)
Experimental result
Intro(4)
Algorithm(3)
Experiment(3/4)
Glass (n=1.5)
10 loop
Here is an another result. the target object is a heart-shaped glass, and the refractive index was 1.5. the assumptions we used is the same as those used in the previous experiment. the result of old method could not estimate the shape accurately, while our method estimated the shape more accurate. these images are the result after 10 iteration.
Initial (Miyazaki 2004)

28. Inverse polarization raytracing(11/11)
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(11/11)
Conclusion(2)
Related work
Intro(4)
Algorithm(3)
Experiment(4/4)
Shape estimation of transparent object
Murase 1992, Hata et al. 1996, Ohara et al. 2003, Ben-Ezra & Nayar 2003, Kutulakos 2005, Saito et al. 1999, Miyazaki et al. 2002, Miyazaki et al. 2004
Shape from polarization
Koshikawa & Shirai 1987, Wolff & Boult 1991, Rahmann 1999, Rahmann 2000, Rahmann & Canterakis 2001, Rahmann 2003, Drbohlav & Sara 2001, Miyazaki et al. 2003
some researchers proposed the methods to estimate the shape of transparent object. but these methods cannot estimate the geometry with general shape. our proposed method estimates the general kind of shapes of transparent objects. some researchers proposed the methods to estimate the shape by using polarization. but these methods only estimate the shape of opaque objects. our proposed method estimates the shape of transparent objects.
Estimate arbitrary shape
Target is transparent
[Our method]

29. Conclusion

30. Summary Shape Iteration min PLZT ND/LP filter Stokes vector Voltage
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(11)
Conclusion(1/2)
Summary
[PLZT polarization camera]
[Inverse polarization raytracing]
Shape
Iteration
min 2
Input
Calc.
PLZT
ND/LP filter
Stokes vector
Voltage
the first topic of my talk was about the plzt polarization camera. by changing the voltage and by using the nd filter and the linear polarizer, the system obtained the stokes vector of observed light. the second topic of my talk was about the inverse polarization raytracing. by minimizing the difference between the input polarization data and the calculated polarization data, the method estimated the shape of transparent object.

31. Future work Improve the accuracy ? Estimating refractive index
Introduction(2)
PLZT polarization camera(12)
Inverse polarization raytracing(11)
Conclusion(2/2)
Future work
[PLZT polarization camera]
[Inverse polarization raytracing]
Improve the accuracy
Realtime measurement
Estimating refractive index ?
the future work is to improve the accuracy of these methods. we also planning to develop a realtime measurement system, and we also planning to develop a method to estimate the refractive index of transparent object. this is the end of my talk. thank you very much.

32. Daisuke Miyazaki 2005 Creative Commons Attribution 4
Daisuke Miyazaki 2005 Creative Commons Attribution 4.0 International License.
Daisuke Miyazaki, Noriyuki Takashima, Akira Yoshida, Eiki Harashima, Katsushi Ikeuchi, "Polarization-based Shape Estimation of Transparent Objects by Using Raytracing and PLZT Camera," in Proceedings of SPIE (Polarization Science and Remote Sensing II, Part of SPIE's International Symposium on Optics and Photonics 2005), Vol. 5888, pp. 1-14, San Diego, CA USA,