Determining Shapes of Transparent Objects from Two Polarization Images Daisuke Miyazaki Masataka Kagesawa Katsushi Ikeuchi The University of Tokyo, Japan Ohayougozaimasu. Good morning. My name is Daisuke Miyazaki from The University of Tokyo, Japan. I would like to give a talk with a title “determining shapes of transparent objects from two polarization images”. December 11, 2002 MVA2002

Modeling transparent objects Polarization-based vision system Unambiguous determination of surface normal using geometrical invariant Transparent object VR Recently, many techniques for modeling objects through observation have been developed. Such modeling has a wide range of applications, including virtual reality and object recognition. Most of these techniques are designed to obtain the shapes of opaque surfaces. Today, I will be talking about our novel method to measure the surface shape of transparent objects. By measuring the degree of polarization of the light reflected from an object surface, we can calculate the surface normal of the object. But the relationship between degree of polarization and surface normal is not 1 to 1. So, we have to solve this ambiguity problem in order to obtain the true surface normal. Our method addresses the ambiguity problem by using the geometrical property of the object surface. December 11, 2002 MVA2002

Related works Wolff 1990 Wolff et al. 1991 Koshikawa 1979 Koshikawa et al. 1987 Not search corresponding points Need many light sources DOP Spherical diffuser Optimization method Saito et al. 1999 Searching corresponding points Not solve ambiguity problem Thermal radiation Binocular stereo Polarization is useful to estimate the shape of specular objects. (click) Koshikawa illuminated specular objects from many directions and picked up a model from a database, which best matches the observed object. (click) Wolff tried to determine the surface shape of objects using a method that combines polarization analysis and binocular stereo. However he did not propose a method to search the corresponding points of two images. (click) Saito et al. tried to measure the surface shape of transparent objects. The drawback of this method is the presence of ambiguity in determining the true surface normal. (click) Rahmann et al. computed the surface shape of specular objects by the optimization method. (click) In our previous work, we have extended Saito’s method and solved the ambiguity problem from a polarization analysis of thermal radiation, and also determined the surface shape of transparent objects. (click) Now, we propose a novel method to determine the surface shape of transparent objects by solving the corresponding problem of binocular stereo in Wolff’s method using the geometrical properties of the object surface. (click) Compared to our previous method, we address the ambiguity problem of surface normal in Saito’s method by rotating the object at a small angle. (click) Our method does not require camera calibration, so the rotation angle for stereo as in Rahmann’s method is not necessary to know. (click) It also does not need any extra infrared camera, such as that used in our previous method. Rahmann et al. 2001 Miyazaki et al. 2002 Not need camera calibration Not need infrared camera Our method December 11, 2002 MVA2002

Outline Rotate the object Target object DOP (Degree Of Polarization) images Region segmentation Search corresponding points 3D model Outline of our method is shown in this slide. First, our measurement system obtains the degree of polarization of the reflected light. Because one measurement of the degree of polarization results in ambiguities, the degree of polarization of a slightly rotated view is employed additionally. Then, we segment both polarization images and divide into some regions. For each region, we search corresponding points. By comparing the degree of polarization at the corresponding points, we determine the surface normal of the object surface. Finally, the 3D geometrical shape of the object surface can be straightforwardly obtained. December 11, 2002 MVA2002

Polarization DOP Origin Unpolarized light Sunlight / incandescent light Perfectly polarized light 1 The light transmitted the polarizer Partially polarized light 0~1 The light hit the object surface DOP(Degree Of Polarization): the ratio of how much the light polarized Object Air Incident light Reflected light Surface normal Transmitted light q Incident angle Reflection Polarizer Light source There are 3 types of light: Unpolarized light, partially polarized light, and perfectly polarized light. (click) The first type of light, unpolarized light, is defined as a light that oscillates uniformly in all direction. Sunlight and incandescent light is unpolarized light. The ratio of how much the light polarized is called the degree of polarization, or shortly, DOP. DOP is 0 for unpolarized light. (click) The second type of light, perfectly polarized light, is defined as a light which oscillates only in one direction. Light that passed through a polarizer is perfectly polarized light. DOP is 1 for this kind of light. (click) The last type of light, partially polarized light, is the intermediate state of unpolarized light and perfecly polarized light. Light that hit an object surface is partially polarized light. DOP ranges from 0 to 1 for this kind of light. Unpolarized light (DOP 0) Pefectly polarized light (DOP 1) Partially polarized light (DOP 0~1) December 11, 2002 MVA2002

Observation Surface normal qP Reflection angle Camera Surface normal qQ Reflection angle Polarizer qP Incident angle Light source qQ Incident angle If we assume the surface of a transparent object as a pure specular surface, then light will hit the object in perfect mirror direction. (click) By knowing the reflection angle and the phase angle, we can determine the surface normal of the object. (click) Also, if we illuminate the object from all directions, then the entire surface shape of the object can be computed. It’s possible to observe the DOP (Degree Of Polarization) by the polarizer-mounted CCD camera located at the top of the object. The DOP depends on the reflection angle, in turn, if we obtain the DOP, we can estimate the reflection angle. Surface normal can be represented by zenith angle and azimuth angle. Reflection angle is equivalent to zenith angle, and phase angle is equivalent to azimuth angle. Thus, if we determine both reflection angle and phase angle, then the surface normal can be determined. Object Light source P Q P Phase angle Q Phase angle December 11, 2002 MVA2002

Ambiguity of phase angle  255 Intensity IminP 1P 2P 360 Phase angle Azimuth angle  -ambiguity This graph expresses how to determine the phase angle, phi. (click) From the intensity observed by the camera, we will determine two candidates of phase angles. (click) We assume the object is a smooth closed object. So, we can determine the phase angle at the occluding boundary. Then, we propagate the determination of the phase angle from occluding boundary to the inner area of the object region. Note that some kind of object, such as basins, or in other words, dimples, cannot be determined by this procedure. Determination of phase angle  Propagate the determination from occluding boundary to the inner area (Assume C2 surface) December 11, 2002 MVA2002

Ambiguity of reflection angle  1 DOP (Degree Of Polarization)  P Brewster angle B 1P 2P 90 Reflection angle Zenith angle  -ambiguity The relation between the reflection angle and the DOP (Degree Of Polarization) is shown in this slide. DOP depends on both the reflection angle and the refractive index of the object. We assume the refractive index is known. DOP will be 0 for 0 degree and 90 degrees. (click) The angle where DOP is 1 is called Brewster angle. (click) Except for Brewster angle, two angles are calculated from one DOP. (click) In the next slides, I will explain the method to solve this ambiguity problem to determine the true angle. Determination of reflection angle  Explain in the following slides December 11, 2002 MVA2002

Rotate the object at a small angle Object rotation Rotate the object at a small angle Solve the ambiguity from two DOP images taken from two directions Rotate Camera Object This method uses two DOP images taken from two different views. We set the camera fix, rotate the object at a small angle, and obtain two DOP images both before and after rotation. By comparing two DOPs, we can obtain the true surface normal of the object. But we have to compare two DOPs in a certain point on the object surface, so that we have to search a pair of points which corresponds to an identical point on the object surface. December 11, 2002 MVA2002

Region segmentation Divide DOP image with curves of 1 DOP (Brewster angle) Region segmentation DOP image DOP 1: white DOP 0: black Result of region segmentation Divided into 3 regions Measure DOP of the object We divide the DOP image into some regions whose boundary will be a curve where DOP equals to 1, namely, Brewster angle. We call the boundary of each region, a Brewster curve. In this picture, DOP image is divided into 3 regions after the region segmentation. Such segmentation makes pointwise disambiguation problem into regionwise disambiguation problem. December 11, 2002 MVA2002

B: Brewster curve N: North pole E: Equator F: Folding curve Gauss’ map N B-E region B-B region B-N region N or E F B B: Brewster curve N: North pole E: Equator F: Folding curve In this slide, I will show that, by considering the Gaussian mapping of the object region onto the Gaussian sphere, we can classify the object region into three types: B-E region shown in the left, B-B region shown in the middle, and B-N region shown in the right. (laser pointer) This curve is the Brewster curve - boundary of each region - , and this curve is the equator - occluding boundary of the object - , and this point is the north pole – surface normal heading to the camera -. We call this curve, a folding curve. B-E region is enclosed by Brewster curve and equator. B-N region is enclosed by Brewster curve and includes north pole. B-B region is enclosed by Brewster curve and folding curve, and does not include equator nor northpole. December 11, 2002 MVA2002

B-E region & B-N region B-E region (qB<q<90o) Definition: A region enclosed by occluding boundaries Determine the occluding boundary from background subtraction B-N region (0o<q<qB) Definition: A region where a point of 0o is included q is 0o or 90o when DOP is 0 Assume there is no self-occlusion, so q is 0o when DOP is 0 We define B-E region as a region enclosed by Brewster curves and occluding boundaries. Occluding boundary is determined by background subtraction. We assume that there is no self-occlusion. From this assumption, reflection angle, theta, will be 0 when DOP is 0. We define B-N region as a region where a point of 0 degree is included; a region where a point of 0 DOP is included. December 11, 2002 MVA2002

B-B region B-B region (0o<q<qB or qB<q<90o) Definition: A region which is not the previous two Apply the following disambiguation method to this region The disambiguation of B-B region is harder than that of B-E region and B-N region. I will explain the disambiguation method of B-B region in the following slides. December 11, 2002 MVA2002

Folding curve A curve (on G) that is a part of the boundary of the region (on G) and is not a Brewster curve (on G) is called a folding curve (on G) Folding curve Brewster curve Equator North pole Gaussian sphere G=Gaussian sphere We define the curve where DOP is 1 as Brewster curve. B-B region on Gaussian sphere is enclosed by Brewster curve and another curve, which we call folding curve. December 11, 2002 MVA2002

Parabolic curve Theorem: Folding curve is parabolic curve Parabolic curve = a curve where Gaussian curvature is 0  Folding curve = geometrical invariant North pole Folding curve Equator Gaussian sphere Object surface We have proven that folding curve on object surface will be a parabolic curve, a curve where Gaussian curvature equals to 0. Parabolic curve is a geometrical characteristic of the object surface which is invariant to the rotation of the object. So, a folding curve helps to search corresponding points. December 11, 2002 MVA2002

Corresponding point Corresponding point  folding curve  great circle  arg min DOP, s. t. surface normal // rotation plane [= rotation direction] point Corresponding point Rotate the object North side Corresponding point Rotate the object South side We define a corresponding point on Gaussian sphere. We use the corresponding point as an intersection of folding curve and great circle which represents the rotation direction. Note that DOP is 1 at Brewster curve and DOP is 0 at equator and north pole. So, the corresponding point will have the minimum DOP through the points on the great circle. To find corresponding point, first we pick up the points where surface normal is parallel to the rotation direction, and then select a point whose DOP is minimum. December 11, 2002 MVA2002

Difference of DOP Compare two DOPs at the pair of corresponding points Derivative of DOP DOP + – 1 qB 90 Compare two DOPs at the pair of corresponding points DOP before rotation DOP after rotation Rotation angle Derivative of DOP We assume the object is rotated at a very small angle, delta theta. Such rotation angle, delta theta, depends on the angle of object rotation and the phase angle. We also assume that we know which direction we rotated the object; but we do not need to know the absolute value of the angle of object rotation. Once phase angle is determined, we can know the sign of the rotation angle, delta theta. The difference of two DOPs is calculated from two DOP images at the pair of corresponding points. So, we know the sign of the derivative of DOP. The derivative of DOP is positive for a smaller angle than Brewster angle, and negative for a larger angle than Brewster angle. Thus, by knowing the sign of the derivative of DOP, we can solve the ambiguity problem. December 11, 2002 MVA2002

Acquisition system Camera Light Polarizer Optical diffuser Object This diagram represents the acquisition system. Object is placed inside the plastic white sphere, which is illuminated by three unpolarized light sources. This sphere diffusely transmits the light and acts as a spherical light source, and then illuminates the entire object surface. There is a hole on top of the sphere and the object is observed by the polarizer-mounted camera placed over the sphere. December 11, 2002 MVA2002

Precision Plastic transparent hemisphere [diameter 3cm] Estimated shape DOP r Result of region segmentation Error DOP 0.17 Reflection angle 8.5 Height 2.6mm Before applying our disambiguation method, we checked the precision of our acquisition system. We measured a plastic transparent hemisphere whose diameter is 3cm. Left upper image is the result of the region segmentation. There are two regions, one is B-E region and the other is B-N region. Estimated shape is rendered in the right upper image. Left lower table shows the error. We, first, calculate the difference between the ground truth and the estimated value at each point, and then, calculate the average value of the absolute value of such difference. This average value is denoted as error in this table. Right lower graph represents the relation between the reflection angle and DOP. Solid line represents the true value and dots represent the obtained value. These results imply that the measurement is not so accurate. This inaccuracy is probably caused by interreflection; the light reflects and refracts inside the object many times. Our method assumes there is no interreflection. Reflection angle q Error (Average absolute difference) Graph of DOP December 11, 2002 MVA2002

Target object Photo [Acrylic bell-shaped object] December 11, 2002 We applied our disambiguation method to an acrylic bell-shaped object shown in this photo. December 11, 2002 MVA2002

DOP images DOP image when the object is not rotated DOP image when the object is rotated at a small angle DOP0:white DOP1:black Rotation direction This slide shows the calculated DOP images. The left is the image taken before the object is rotated and the right is the image taken after the object is rotated at approximately 8 degrees. The rotation direction is expressed as an arrow in this slide. DOP images are expressed in gray image, where white represents 0 DOP and black represents 1 DOP. We rotate the object about 8° December 11, 2002 MVA2002

Region segmentation result Result of region segmentation when the object is not rotated Result of region segmentation when the object is rotated at a small angle Rotation direction After obtaining DOP images, we divided the images into some regions. The left image is the result of region segmentation computed from the DOP image taken before the object is rotated. There are three regions: blue region is B-E region, yellow region is B-B region, and brown region is B-N region. The right image is the result of region segmentation computed from the DOP image taken after the object is rotated. We rotate the object about 8° December 11, 2002 MVA2002

Disambiguation of B-B region Negative Positive Surface normal was  Rotation direction was  0.089 0.084 Derivative of DOP Positive Negative qB 90 We searched a pair of corresponding points at the B-B region. At that point, phase angle told us that the surface normal is heading the opposite as the rotation direction. Thus, the sign of the rotation angle, delta theta, is determined to be negative. DOP at the corresponding point for the object before rotation was 0.089, and DOP for the object after rotation was 0.084. Thus, the sign of the difference of DOP is determined to be negative. As a result, the derivative of DOP become positive, that is, the angle of the B-B region is smaller than Brewster angle. December 11, 2002 MVA2002

Rendered image Shading image December 11, 2002 MVA2002 The distribution of surface normal is integrated into the distribution of height. This slide shows the rendered image of the estimated shape of the object. December 11, 2002 MVA2002

Rendered image Photo Raytracing image December 11, 2002 MVA2002 The right image is another rendered image of the estimated shape. The image is rendered by raytracer. For comparison, the object’s real image is shown in the left image. December 11, 2002 MVA2002

Error Comparison of true value and estimated value The diameter(width) of the object is 24mm Error is 0.4mm (Average of the difference of the height) To evaluate the precision of our method, we compared the true value and the estimated value. Note that the true value is artificially created by hand from the photograph taken from the side of the object. The true height is expressed as a solid line, and the estimated height is expressed as dots. We calculated the difference between the true height and the estimated height, and calculated the average of the absolute value of such difference. Such average value was 0.4mm, where the width of the object was 24mm. True value is made by hand December 11, 2002 MVA2002

Conclusions A method to measure the surface shape of transparent object based on the analysis of polarization and geometrical characteristics Determined the surface normal with no ambiguity Detected a pair of corresponding points of transparent surface Determined the surface normal of the entire surface at once Measured a transparent object which is not a hemisphere We proposed a method to measure the surface shape of transparent object by vision-based method based on the analysis of polarization and geometrical characteristics. December 11, 2002 MVA2002

Future works Higher precision (dealing with interreflections) Estimation of refractive index More elegant method for determining phase angle  Our future works are to deal with interreflections and to acquire the shape of transparent objects in higher precision. We plan to develop a method to estimate the shape and the refractive index at the same time. Besides that, we also try to propose more elegant method to determine the phase angle. Arigatougozaimashita. Thank you very much. December 11, 2002 MVA2002

Daisuke Miyazaki 2002 Creative Commons Attribution 4 Daisuke Miyazaki 2002 Creative Commons Attribution 4.0 International License. http://www.cvl.iis.u-tokyo.ac.jp/ D. Miyazaki, M. Kagesawa, K. Ikeuchi, "Determining Shapes of Transparent Objects from Two Polarization Images," in Proceedings of IAPR Workshop on Machine Vision Applications, pp.26-31, Nara, Japan, 2002.12 December 11, 2002 MVA2002