1. Determining Shapes of Transparent Objects from Two Polarization Images
Daisuke Miyazaki
Masataka Kagesawa
Katsushi Ikeuchi
The University of Tokyo, Japan
Ohayougozaimasu. Good morning. My name is Daisuke Miyazaki from The University of Tokyo, Japan.
I would like to give a talk with a title “determining shapes of transparent objects from two polarization images”.
December 11, 2002
MVA2002

2. Modeling transparent objects
Polarization-based vision system
Unambiguous determination of surface normal using geometrical invariant
Transparent object VR
Recently, many techniques for modeling objects through observation have been developed. Such modeling has a wide range of applications, including virtual reality and object recognition. Most of these techniques are designed to obtain the shapes of opaque surfaces.
Today, I will be talking about our novel method to measure the surface shape of transparent objects. By measuring the degree of polarization of the light reflected from an object surface, we can calculate the surface normal of the object. But the relationship between degree of polarization and surface normal is not 1 to 1. So, we have to solve this ambiguity problem in order to obtain the true surface normal. Our method addresses the ambiguity problem by using the geometrical property of the object surface.
December 11, 2002
MVA2002

3. Related works Wolff 1990 Wolff et al. 1991
Koshikawa 1979 Koshikawa et al. 1987
Not search corresponding points
Need many light sources
DOP
Spherical diffuser
Optimization method
Saito et al. 1999
Searching corresponding points
Not solve ambiguity problem
Thermal radiation
Binocular stereo
Polarization is useful to estimate the shape of specular objects.
(click) Koshikawa illuminated specular objects from many directions and picked up a model from a database, which best matches the observed object. (click) Wolff tried to determine the surface shape of objects using a method that combines polarization analysis and binocular stereo. However he did not propose a method to search the corresponding points of two images. (click) Saito et al. tried to measure the surface shape of transparent objects. The drawback of this method is the presence of ambiguity in determining the true surface normal.
(click) Rahmann et al. computed the surface shape of specular objects by the optimization method. (click) In our previous work, we have extended Saito’s method and solved the ambiguity problem from a polarization analysis of thermal radiation, and also determined the surface shape of transparent objects.
(click) Now, we propose a novel method to determine the surface shape of transparent objects by solving the corresponding problem of binocular stereo in Wolff’s method using the geometrical properties of the object surface. (click) Compared to our previous method, we address the ambiguity problem of surface normal in Saito’s method by rotating the object at a small angle.
(click) Our method does not require camera calibration, so the rotation angle for stereo as in Rahmann’s method is not necessary to know. (click) It also does not need any extra infrared camera, such as that used in our previous method.
Rahmann et al. 2001
Miyazaki et al. 2002
Not need camera calibration
Not need infrared camera
Our method
December 11, 2002
MVA2002

4. Outline Rotate the object Target object
DOP (Degree Of Polarization) images
Region segmentation
Search corresponding points
3D model
Outline of our method is shown in this slide.
First, our measurement system obtains the degree of polarization of the reflected light. Because one measurement of the degree of polarization results in ambiguities, the degree of polarization of a slightly rotated view is employed additionally. Then, we segment both polarization images and divide into some regions. For each region, we search corresponding points. By comparing the degree of polarization at the corresponding points, we determine the surface normal of the object surface. Finally, the 3D geometrical shape of the object surface can be straightforwardly obtained.
December 11, 2002
MVA2002

5. Polarization DOP Origin Unpolarized light
Sunlight / incandescent light
Perfectly polarized light 1
The light transmitted the polarizer
Partially polarized light
0~1
The light hit the object surface
DOP(Degree Of Polarization): the ratio of how much the light polarized
Object
Air
Incident light
Reflected light
Surface normal
Transmitted light q
Incident
angle
Reflection
Polarizer
Light source
There are 3 types of light: Unpolarized light, partially polarized light, and perfectly polarized light.
(click) The first type of light, unpolarized light, is defined as a light that oscillates uniformly in all direction. Sunlight and incandescent light is unpolarized light.
The ratio of how much the light polarized is called the degree of polarization, or shortly, DOP.
DOP is 0 for unpolarized light.
(click) The second type of light, perfectly polarized light, is defined as a light which oscillates only in one direction. Light that passed through a polarizer is perfectly polarized light. DOP is 1 for this kind of light.
(click) The last type of light, partially polarized light, is the intermediate state of unpolarized light and perfecly polarized light. Light that hit an object surface is partially polarized light. DOP ranges from 0 to 1 for this kind of light.
Unpolarized light
(DOP 0)
Pefectly polarized light (DOP 1)
Partially polarized light
(DOP 0~1)
December 11, 2002
MVA2002

6. Observation Surface normal qP Reflection angle Camera Surface normalqQ
Reflection angle
Polarizer qP
Incident angle
Light
source qQ
Incident angle
If we assume the surface of a transparent object as a pure specular surface, then light will hit the object in perfect mirror direction. (click) By knowing the reflection angle and the phase angle, we can determine the surface normal of the object. (click) Also, if we illuminate the object from all directions, then the entire surface shape of the object can be computed.
It’s possible to observe the DOP (Degree Of Polarization) by the polarizer-mounted CCD camera located at the top of the object. The DOP depends on the reflection angle, in turn, if we obtain the DOP, we can estimate the reflection angle.
Surface normal can be represented by zenith angle and azimuth angle. Reflection angle is equivalent to zenith angle, and phase angle is equivalent to azimuth angle. Thus, if we determine both reflection angle and phase angle, then the surface normal can be determined.
Object
Light source P Q P
Phase angle Q
Phase angle
December 11, 2002
MVA2002

7. Ambiguity of phase angle 
255
Intensity
IminP
1P
2P
360
Phase angle Azimuth angle 
-ambiguity
This graph expresses how to determine the phase angle, phi. (click) From the intensity observed by the camera, we will determine two candidates of phase angles. (click)
We assume the object is a smooth closed object. So, we can determine the phase angle at the occluding boundary. Then, we propagate the determination of the phase angle from occluding boundary to the inner area of the object region. Note that some kind of object, such as basins, or in other words, dimples, cannot be determined by this procedure.
Determination of phase angle 
Propagate the determination from occluding boundary to the inner area (Assume C2 surface)
December 11, 2002
MVA2002

8. Ambiguity of reflection angle 1
DOP (Degree Of Polarization)  P
Brewster angle B
1P
2P
90
Reflection angle
Zenith angle 
-ambiguity
The relation between the reflection angle and the DOP (Degree Of Polarization) is shown in this slide.
DOP depends on both the reflection angle and the refractive index of the object. We assume the refractive index is known.
DOP will be 0 for 0 degree and 90 degrees. (click) The angle where DOP is 1 is called Brewster angle. (click) Except for Brewster angle, two angles are calculated from one DOP. (click) In the next slides, I will explain the method to solve this ambiguity problem to determine the true angle.
Determination of reflection angle 
Explain in the following slides
December 11, 2002
MVA2002

9. Rotate the object at a small angle
Object rotation
Rotate the object at a small angle
Solve the ambiguity from two DOP images taken from two directions
Rotate
Camera
Object
This method uses two DOP images taken from two different views. We set the camera fix, rotate the object at a small angle, and obtain two DOP images both before and after rotation. By comparing two DOPs, we can obtain the true surface normal of the object. But we have to compare two DOPs in a certain point on the object surface, so that we have to search a pair of points which corresponds to an identical point on the object surface.
December 11, 2002
MVA2002

10. Region segmentation
Divide DOP image with curves of 1 DOP (Brewster angle)
Region segmentation
DOP image
DOP 1: white
DOP 0: black
Result of
region segmentation
Divided into
3 regions
Measure DOP of the object
We divide the DOP image into some regions whose boundary will be a curve where DOP equals to 1, namely, Brewster angle. We call the boundary of each region, a Brewster curve. In this picture, DOP image is divided into 3 regions after the region segmentation.
Such segmentation makes pointwise disambiguation problem into regionwise disambiguation problem.
December 11, 2002
MVA2002

11. B: Brewster curve N: North pole E: Equator F: Folding curve
Gauss’ map N
B-E region
B-B region
B-N region N or E F B
B: Brewster curve N: North pole E: Equator F: Folding curve
In this slide, I will show that, by considering the Gaussian mapping of the object region onto the Gaussian sphere, we can classify the object region into three types: B-E region shown in the left, B-B region shown in the middle, and B-N region shown in the right.
(laser pointer) This curve is the Brewster curve - boundary of each region - , and this curve is the equator - occluding boundary of the object - , and this point is the north pole – surface normal heading to the camera -. We call this curve, a folding curve.
B-E region is enclosed by Brewster curve and equator. B-N region is enclosed by Brewster curve and includes north pole. B-B region is enclosed by Brewster curve and folding curve, and does not include equator nor northpole.
December 11, 2002
MVA2002

12. B-E region & B-N region B-E region (qB<q<90o)
Definition: A region enclosed by occluding boundaries
Determine the occluding boundary from background subtraction
B-N region (0o<q<qB)
Definition: A region where a point of 0o is included
q is 0o or 90o when DOP is 0
Assume there is no self-occlusion, so q is 0o when DOP is 0
We define B-E region as a region enclosed by Brewster curves and occluding boundaries. Occluding boundary is determined by background subtraction.
We assume that there is no self-occlusion. From this assumption, reflection angle, theta, will be 0 when DOP is 0. We define B-N region as a region where a point of 0 degree is included; a region where a point of 0 DOP is included.
December 11, 2002
MVA2002

13. B-B region B-B region (0o<q<qB or qB<q<90o)
Definition: A region which is not the previous two
Apply the following disambiguation method to this region
The disambiguation of B-B region is harder than that of B-E region and B-N region. I will explain the disambiguation method of B-B region in the following slides.
December 11, 2002
MVA2002

14. Folding curve
A curve (on G) that is a part of the boundary of the region (on G) and is not a Brewster curve (on G) is called a folding curve (on G)
Folding curve
Brewster curve
Equator
North pole
Gaussian sphere
G=Gaussian sphere
We define the curve where DOP is 1 as Brewster curve. B-B region on Gaussian sphere is enclosed by Brewster curve and another curve, which we call folding curve.
December 11, 2002
MVA2002

15. Parabolic curve Theorem: Folding curve is parabolic curve
Parabolic curve = a curve where Gaussian curvature is 0
 Folding curve = geometrical invariant
North pole
Folding curve
Equator
Gaussian sphere
Object surface
We have proven that folding curve on object surface will be a parabolic curve, a curve where Gaussian curvature equals to 0. Parabolic curve is a geometrical characteristic of the object surface which is invariant to the rotation of the object. So, a folding curve helps to search corresponding points.
December 11, 2002
MVA2002

16. Corresponding point
Corresponding point  folding curve  great circle  arg min DOP, s. t surface normal // rotation plane
[= rotation direction]
point
Corresponding
point
Rotate the object
North side
Corresponding
point
Rotate the object
South side
We define a corresponding point on Gaussian sphere.
We use the corresponding point as an intersection of folding curve and great circle which represents the rotation direction. Note that DOP is 1 at Brewster curve and DOP is 0 at equator and north pole. So, the corresponding point will have the minimum DOP through the points on the great circle. To find corresponding point, first we pick up the points where surface normal is parallel to the rotation direction, and then select a point whose DOP is minimum.
December 11, 2002
MVA2002

17. Difference of DOP Compare two DOPs at the pair of corresponding points
Derivative of DOP
DOP + – 1 qB
90
Compare two DOPs at the pair of corresponding points
DOP before rotation
DOP after rotation
Rotation angle
Derivative of DOP
We assume the object is rotated at a very small angle, delta theta. Such rotation angle, delta theta, depends on the angle of object rotation and the phase angle. We also assume that we know which direction we rotated the object; but we do not need to know the absolute value of the angle of object rotation. Once phase angle is determined, we can know the sign of the rotation angle, delta theta.
The difference of two DOPs is calculated from two DOP images at the pair of corresponding points. So, we know the sign of the derivative of DOP. The derivative of DOP is positive for a smaller angle than Brewster angle, and negative for a larger angle than Brewster angle. Thus, by knowing the sign of the derivative of DOP, we can solve the ambiguity problem.
December 11, 2002
MVA2002

18. Acquisition system Camera Light Polarizer Optical diffuser Object
This diagram represents the acquisition system.
Object is placed inside the plastic white sphere, which is illuminated by three unpolarized light sources. This sphere diffusely transmits the light and acts as a spherical light source, and then illuminates the entire object surface. There is a hole on top of the sphere and the object is observed by the polarizer-mounted camera placed over the sphere.
December 11, 2002
MVA2002

19. Precision Plastic transparent hemisphere [diameter 3cm]
Estimated shape
DOP r
Result of region segmentation
Error
DOP
0.17
Reflection angle
8.5
Height
2.6mm
Before applying our disambiguation method, we checked the precision of our acquisition system. We measured a plastic transparent hemisphere whose diameter is 3cm.
Left upper image is the result of the region segmentation. There are two regions, one is B-E region and the other is B-N region.
Estimated shape is rendered in the right upper image.
Left lower table shows the error. We, first, calculate the difference between the ground truth and the estimated value at each point, and then, calculate the average value of the absolute value of such difference. This average value is denoted as error in this table.
Right lower graph represents the relation between the reflection angle and DOP. Solid line represents the true value and dots represent the obtained value.
These results imply that the measurement is not so accurate. This inaccuracy is probably caused by interreflection; the light reflects and refracts inside the object many times. Our method assumes there is no interreflection.
Reflection angle q
Error (Average absolute difference)
Graph of DOP
December 11, 2002
MVA2002

20. Target object Photo [Acrylic bell-shaped object] December 11, 2002
We applied our disambiguation method to an acrylic bell-shaped object shown in this photo.
December 11, 2002
MVA2002

21. DOP images DOP image when the object is not rotated
DOP image when the object is rotated at a small angle
DOP0:white DOP1:black
Rotation direction
This slide shows the calculated DOP images. The left is the image taken before the object is rotated and the right is the image taken after the object is rotated at approximately 8 degrees. The rotation direction is expressed as an arrow in this slide. DOP images are expressed in gray image, where white represents 0 DOP and black represents 1 DOP.
We rotate the object about 8°
December 11, 2002
MVA2002

22. Region segmentation result
Result of region segmentation when the object is not rotated
Result of region segmentation when the object is rotated at a small angle
Rotation direction
After obtaining DOP images, we divided the images into some regions.
The left image is the result of region segmentation computed from the DOP image taken before the object is rotated. There are three regions: blue region is B-E region, yellow region is B-B region, and brown region is B-N region.
The right image is the result of region segmentation computed from the DOP image taken after the object is rotated.
We rotate the object about 8°
December 11, 2002
MVA2002

23. Disambiguation of B-B region
Negative
Positive
Surface normal was  Rotation direction was 
0.089
0.084
Derivative of DOP
Positive
Negative qB
90
We searched a pair of corresponding points at the B-B region. At that point, phase angle told us that the surface normal is heading the opposite as the rotation direction. Thus, the sign of the rotation angle, delta theta, is determined to be negative. DOP at the corresponding point for the object before rotation was 0.089, and DOP for the object after rotation was Thus, the sign of the difference of DOP is determined to be negative. As a result, the derivative of DOP become positive, that is, the angle of the B-B region is smaller than Brewster angle.
December 11, 2002
MVA2002

24. Rendered image Shading image December 11, 2002 MVA2002
The distribution of surface normal is integrated into the distribution of height.
This slide shows the rendered image of the estimated shape of the object.
December 11, 2002
MVA2002

25. Rendered image Photo Raytracing image December 11, 2002 MVA2002
The right image is another rendered image of the estimated shape. The image is rendered by raytracer. For comparison, the object’s real image is shown in the left image.
December 11, 2002
MVA2002

26. Error Comparison of true value and estimated value
The diameter(width) of the object is 24mm
Error is 0.4mm (Average of the difference of the height)
To evaluate the precision of our method, we compared the true value and the estimated value. Note that the true value is artificially created by hand from the photograph taken from the side of the object.
The true height is expressed as a solid line, and the estimated height is expressed as dots.
We calculated the difference between the true height and the estimated height, and calculated the average of the absolute value of such difference. Such average value was 0.4mm, where the width of the object was 24mm.
True value is made by hand
December 11, 2002
MVA2002

27. Conclusions
A method to measure the surface shape of transparent object based on the analysis of polarization and geometrical characteristics
Determined the surface normal with no ambiguity
Detected a pair of corresponding points of transparent surface
Determined the surface normal of the entire surface at once
Measured a transparent object which is not a hemisphere
We proposed a method to measure the surface shape of transparent object by vision-based method based on the analysis of polarization and geometrical characteristics.
December 11, 2002
MVA2002

28. Future works Higher precision (dealing with interreflections)
Estimation of refractive index
More elegant method for determining phase angle 
Our future works are to deal with interreflections and to acquire the shape of transparent objects in higher precision. We plan to develop a method to estimate the shape and the refractive index at the same time. Besides that, we also try to propose more elegant method to determine the phase angle.
Arigatougozaimashita. Thank you very much.
December 11, 2002
MVA2002

29. Daisuke Miyazaki 2002 Creative Commons Attribution 4
Daisuke Miyazaki 2002 Creative Commons Attribution 4.0 International License.
D. Miyazaki, M. Kagesawa, K. Ikeuchi, "Determining Shapes of Transparent Objects from Two Polarization Images," in Proceedings of IAPR Workshop on Machine Vision Applications, pp.26-31, Nara, Japan,
December 11, 2002
MVA2002