What Does Motion Reveal About Transparency ? Moshe Ben-Ezra and Shree K. Nayar Columbia University ICCV Conference October 2003, Nice, France This work was supported by an NSF ITR Award IIS

Transparency is Very Challenging Existence of a transparent object. Finding its shape and pose

Real and Virtual Features Lambertian V1V1 V2V2 F` V2V2   V1V1 F Specular F F`F` V1V1 V2V2 Transparent

Environmental Matting* * Zongker, el al. SIGGRAPH 99, Alternating pattern Object Camera Does not recover shape and pose. Requires controlled environment.

Shape from Polarization in Highlight* * Saito et al. CVPR’99. Object Camera Light Rotating Polarizer Limited to a single interface at the object’s surface. Requires controlled environment. N

Shape from Refraction and Motion* * H. Murase. PAMI, 1992 Camera Water Single interface only. Fixed Pattern

Motion is Key to Transparency

Transparent Shape From Motion Given: Views And a Parametric Model (such as super-ellipse) Recover: Shape: Values of parameters (e, n) Pose: Rotation R, Translation T General analytic solution does not exist.

Transparency From Motion Reversed rays are parallel to each other regardless of the complexity of their paths Distant feature

Approach: Initialization Image Plane

Approach: Initial Guess

Approach: Refine

Error Function (0,0,1) r 1,1.. r 1,n r 2,1.. r 2,n  - Object’s shape parameter vector R,T - Object’s pose

Simulation Setup Parallel rays from features Transparent object Camera side rays

Example (Simulation) Single Parameter. Newton-Raphson optimization Initial Guess Symmetric Superellipse (n=e)

Evaluation (Simulation) GTGT Both init Pos res Sphere Ground Truth Initial Guess Computed Result Shape Error GTBoth InitBoth Res Lens GTBoth InitBoth Res Cube GTBoth Init Both Res Water Pipe

Real Experiment: Sphere

Features

Initial Guess

Setup: Initial Guess Initial Guess: Diameter: 8 inch

Setup: Result Ground Truth: Diameter: 3 inch. Computed: 3.18 inch

Result

Real Test: Water Filled Pipe

Features

Initial Guess

Setup: Initial Guess Initial guess: Diameter: 200.0mm Thickness: 20.0mm

Setup: Result Ground Truth: Diameter: 117.0mm Thickness: 3.0mm Computed: Diameter: 116.1mm Thickness: 2.3mm

Result

Real Test: Superquadric

Features

Initial Guess

Result Ground truth: e = ?Computed: e = 0.18

Summary Shape and pose parameters Multiple interfaces No Segmentation required

Parameterizations of Interest Polynomials: modeling surfaces, lenses CAD models: shape of industrial objects Dynamic models: time dependent parameters

Assumptions Camera parameters are known. Features are far* and are trackable. A proper model and a hypothesis (an initial guess) are given. * One possible assumption.

Real Tests Setup

Implementation Features were manually selected and tracked (9 views). Captured rays, a model, refraction index and a hypothesis were given as inputs. Shape and pose were recovered using simple gradient decent (with derivatives).

The Physics of Transparency First Interface: μ 1 → μ 2 Second Interface: μ 2 →μ 1 33 11 11 33 N1N1 N2N2 22 22

Parametric Shape Examples Super-Ellipse 2 parameters Spherical Harmonics 8 parameters No analytic solution