1. What Does a Single Light-Ray Reveal About a Transparent Object?
Chia-Yin Tsai
Ashok Veeraraghavan
Aswin C. Sankaranarayanan
ICIP 2015
In this paper, we derive depth-normal ambiguities for refractive objects using light ray correspondence.

2. image courtesy: Huang, Juinn-Kai
Transparent objects do not have appearance of their own (pause) they merely distort their surrounding environment. (pause) So how can we find their shape?
image courtesy: Huang, Juinn-Kai

3. Instead of using images, a powerful approach is to infer the shape of the object by studying how it interacts with light rays. (pause)
However, even when we study the problem using light rays, for many objects, light rays will bent at least twice --- once each upon entering and exiting the object. (pause)
This makes the reconstruction of the shape a hard problem!

4. Light ray correspondence?
We assume that we know the refractive index of the object. With the knowledge of an ingoing and outgoing light ray, can we infer the depth and normal of the two refraction events?
Known entities are marked in green.

5. Light ray correspondence
Known entities are marked in green.

6. Depth-normal ambiguity for transparent objects
Theorem 1: Given the surface normal at the first refraction, the depth is unique.
Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
In our work, we find depth-normal ambiguity for transparent objects.
First, given the surface normal at the first refraction location, we show that the depth is unique.
Second, given the surface depth at the first refraction, we show that the normal is constrained to lie on a 1D curve.
In the following, we will explain the geometric interpretation of the two theorems.
Known entities are marked in green.

7. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
First we explain why given the surface normal at the first refraction, the depth is unique.
Assume that we know the surface normal n1 at the first refraction
Known entities are marked in green.

8. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
By Snell’s law, we know how light ray changes its direction after refraction.
Known entities are marked in green.

9. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
That is, we know o1.
Known entities are marked in green.

10. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
However, we do not know where there first refraction happens. Therefore, we can have different hypothesized depth, d1^1,
Known entities are marked in green.

11. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
Known entities are marked in green.

12. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
and d1^3.
Known entities are marked in green.

13. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
We can observe that all possible light rays after first refraction sweep through a plane.
Known entities are marked in green.

14. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
By intersecting the outgoing light ray with this plane, we find the location of second refraction.
Known entities are marked in green.

15. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
Thus, we can find the depth corresponding to this particular location.
Known entities are marked in green.

16. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
Therefore, we can see that for a given normal, the depth is unique.
Known entities are marked in green.

17. Theorem 1: Given the surface normal at the first refraction, the depth is unique.
Notice that we assume that we can find the second refraction location by intersecting the plane with the outgoing light ray. If the light path is coplanar, i.e., completely lies on a plane, we cannot find a unique intersection. In this case, given normal, the depth is still unknown.
Known entities are marked in green.

18. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
Now we explain the second ambiguity. Given the surface depth of the first refraction location, the normal is constrained to lie on a 1D curve.
Given the surface depth, the location of the first refraction, v1, will be determined.
Known entities are marked in green.

19. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
The second refraction will happen along the outgoing light ray.
We hypothesize that it happens at v2^1. Thus, we can know the light ray direction connecting the first and second refraction, o1^1.
Known entities are marked in green.

20. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
Now we know i1 and o1, thus by Snell’s law, we can find the surface normal n1^2 at the first refraction.
Known entities are marked in green.

21. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
Similarly, we can assume the refraction happens at v2^2. We can also calculate the corresponding surface normal n_1^2.
Known entities are marked in green.

22. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
Same procedure holds for v2^3.
Known entities are marked in green.

23. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
All possible light ray after the first refraction will lie on the gray colored plane.
Known entities are marked in green.

24. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
We can represent the plane using orthonormal basis.
Known entities are marked in green.

25. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
Using Snell’s law, we can represent the normal in terms of i1 and o1.
Known entities are marked in green.

26. Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
The normal can be found by intersecting an oblique cone and a unit sphere. We can see that all possible normals corresponding to the depth will trace a 1D curve on the unit sphere.
Known entities are marked in green.

27. Depth-normal ambiguity for transparent objects
Theorem 1: Given the surface normal at the first refraction, the depth is unique.
Theorem 2: Given the surface depth at the first refraction, the normal is constrained to lie on a 1D curve.
Up till this point, we present the geometric interpretation of the two depth-normal ambiguities for transparent objects. The two properties provide insights for sufficient conditions for reconstructing a transparent object.
Known entities are marked in green.

28. At least three light ray correspondences are needed to reconstruct the surface.
For example, we will need at least three light ray correspondences to reconstruct a surface patch. This can be explained by the depth-normal ambiguity.
If we hypothesize a depth, it will give us a set of corresponding normal. We can also see this surface patch from another camera view. And it will also create a set of possible normals. As we can see from the left hand side image, any two curves can potentially intersect, thus it does not give a unique estimate.
However, if we include a third camera view, it can check if all three curves intersect at the same normal. Therefore, we need at least three cameras to reconstruct a transparent object.
Normal estimation from two views.
Any two curves can potentially intersect.
Normal estimation from three views.
A third view is added to check the solution.

29. Single-view reconstruction algorithm for one-side planar transparent objects
If we further impose a surface model, such as the object being planar on one side, we can reconstruct the shape of a refractive object using just a single camera.
The key is to hypothesize different surface normal and see which creates a surface that has a consistent normal with our hypothesis.

30. data courtesy: Kutulakos and Steger, IJCV 2008
Here we show our reconstruction result using the planar assumption. Notice how the details of the different facets are all preserved.
data courtesy: Kutulakos and Steger, IJCV 2008

31. http://imagesci.ece.cmu.edu for more details, visit
For more details, visit our lab webpage.