1. What Does Motion Reveal About Transparency ? Moshe Ben-Ezra and Shree K. Nayar Columbia University ICCV Conference October 2003, Nice, France This work was supported by an NSF ITR Award IIS-00-85864

2. Transparency is Very Challenging Existence of a transparent object. Finding its shape and pose

3. Real and Virtual Features Lambertian V1V1 V2V2 F` V2V2   V1V1 F Specular F F`F` V1V1 V2V2 Transparent

4. Environmental Matting* * Zongker, el al. SIGGRAPH 99, Alternating pattern Object Camera Does not recover shape and pose. Requires controlled environment.

5. Shape from Polarization in Highlight* * Saito et al. CVPR’99. Object Camera Light Rotating Polarizer Limited to a single interface at the object’s surface. Requires controlled environment. N

6. Shape from Refraction and Motion* * H. Murase. PAMI, 1992 Camera Water Single interface only. Fixed Pattern

7. Motion is Key to Transparency

8. Transparent Shape From Motion Given: Views And a Parametric Model (such as super-ellipse) Recover: Shape: Values of parameters (e, n) Pose: Rotation R, Translation T General analytic solution does not exist.

9. Transparency From Motion Reversed rays are parallel to each other regardless of the complexity of their paths Distant feature

10. Approach: Initialization Image Plane

11. Approach: Initial Guess

12. Approach: Refine

13. Error Function (0,0,1) r 1,1.. r 1,n r 2,1.. r 2,n  - Object’s shape parameter vector R,T - Object’s pose

14. Simulation Setup Parallel rays from features Transparent object Camera side rays

15. Example (Simulation) Single Parameter. Newton-Raphson optimization Initial Guess Symmetric Superellipse (n=e)

16. Evaluation (Simulation) GTGT Both init Pos res Sphere Ground Truth Initial Guess Computed Result Shape Error GTBoth InitBoth Res Lens GTBoth InitBoth Res Cube GTBoth Init Both Res Water Pipe

17. Real Experiment: Sphere

18. Features

19. Initial Guess

20. Setup: Initial Guess Initial Guess: Diameter: 8 inch

21. Setup: Result Ground Truth: Diameter: 3 inch. Computed: 3.18 inch

22. Result

23. Real Test: Water Filled Pipe

24. Features

25. Initial Guess

26. Setup: Initial Guess Initial guess: Diameter: 200.0mm Thickness: 20.0mm

27. Setup: Result Ground Truth: Diameter: 117.0mm Thickness: 3.0mm Computed: Diameter: 116.1mm Thickness: 2.3mm

28. Result

29. Real Test: Superquadric

30. Features

31. Initial Guess

32. Result Ground truth: e = ?Computed: e = 0.18

33. Summary Shape and pose parameters Multiple interfaces No Segmentation required

35. Parameterizations of Interest Polynomials: modeling surfaces, lenses CAD models: shape of industrial objects Dynamic models: time dependent parameters

36. Assumptions Camera parameters are known. Features are far* and are trackable. A proper model and a hypothesis (an initial guess) are given. * One possible assumption.

37. Real Tests Setup

38. Implementation Features were manually selected and tracked (9 views). Captured rays, a model, refraction index and a hypothesis were given as inputs. Shape and pose were recovered using simple gradient decent (with derivatives).

39. The Physics of Transparency First Interface: μ 1 → μ 2 Second Interface: μ 2 →μ 1 33 11 11 33 N1N1 N2N2 22 22

40. Parametric Shape Examples Super-Ellipse 2 parameters Spherical Harmonics 8 parameters No analytic solution