Dynamic Refraction Stereo 7. Contributions Refractive disparity optimization gives stable reconstructions regardless of surface shape Require no geometric assumptions other than single light refraction Produces a full resolution height map and a separate, full-resolution normal map Highly-detailed reconstructions for a variety of complex, deforming liquid surfaces Kiriakos N. Kutulakos cs.toronto.edu Nigel J. W. Morris dgp.toronto.edu 8. Experimental results Experiments with dynamic water surfaces t=0.6s Height map Tilt map Instantaneous 3D reconstructions t=0.92s t=1.32s Simulations vs. Ground-truth ExperimentsRefractive index estimation Observed results (solid red) vs simulated results (dotted blue) for 0.08 pixel localization error for set of planar refractive surfaces at various heights Total reconstruction error as function of refractive index (liquid was water) Height map Tilt map Height map Tilt map t=0.92s t=0.3s t=0.91s 4. Refractive Stereo Algorithm 2. Related Work Single-view methods (require extra assumptions/optics): Shape from Distortion (Murase, PAMI 1992)  assumes constant mean distance Shape from Refraction (Jähne et al., JOSA 1994)  uses collimating lens & lighting gradient Multi-view methods: Sanderson et al. (PAMI 1998), Bonfort & Sturm (ICCV 2003)  static mirror scenes (known refractive index),  optimization degrades for shallow liquid heights Multi-media photogrammetry (Flach & Maas, IAPRS 2000)  known parametric shape model 3. The Refraction Stereo Constraint Key insight: For a generic refractive surface, knowledge of pixel q & pattern point C(q) defines a constraint curve in the 3D space of possible normals & refractive index values Height hypothesis in Camera 1  correspondence in Camera 2 q & C(q) in Camera 1  1 st constraint curve q’ & C(q’) in Camera 2  2 nd constraint curve Generically, curves do not intersect for incorrect height hypotheses  can resolve height, normal, refractive index Goal: Reconstruct instantaneous height map & normal map of time-varying refractive surface projecting to 2 cameras Calibrated & synchronized cameras Arbitrary, time-varying surface shape (i.e. no prior shape models) Refractive index unknown Calibrated pattern at known 3D position, visible through liquid Known 1-1 mapping from pixels q to points C(q) on pattern 1. Dynamic Refraction Stereo Problem p n Cam1 (time t) Cam2 (time t) q C(q) Discretize interval of possible refractive index values. For each value, do Steps 1-4: Step 1: Initialize correspondence function C(.) for time 0 Step 2: For each time t and each pixel q in Camera 1 2a: (Refractive disparity optimization) 1D optimization along ray of q, searching for height hypothesis consistent with both viewpoints 2b: (Bundle adjustment) 5D optimization of p and n using reprojection error Step 3: Fuse depth & normal map to obtain 3D surface Step 4: Update correspondence function for next frame Choose refractive index value minimizing total reconstruction error (across all frames and pixels) 5. 1D Optimization: Refractive Disparity Large liquid heights:  both criteria are 0 when p on ‘true’ surface As height → 0  estimation of n 1, n 2 unstable  ||d 1 || 2 + ||d 2 || 2 remains stable (reduces to standard stereo) minimize angle(n 1,n 2 ) p n1n1 n2n2 minimize ||d 1 || 2 + ||d 2 || 2 Bonfort & SturmRefractive Disparity n1n1 n2n2 n2n2 n1n1 d1d1 d2d2 n1n1 n2n2 p n q C(q) q’ C(q’) 6. Computing Correspondence C(q) Iteratively compute flow between un-refracted & refracted views using Lucas-Kanade (Baker & Matthews, IJCV 2004) Refracted view (time t) Un-refracted view (reference) t=0.3st=0.35s t=0.4s t=0.75st=0.83s t=0.91s