Image-based Water Surface Reconstruction with Refractive Stereo Nigel Morris University of Toronto

Motivation Computational Fluid Dynamics are extremely complex and difficult to simulate Why not capture fluid effects from reality? We present the first step to capturing fluids from reality – reconstructing water surfaces May eventually be useful for determining fluid flow

Previous Work Shape from shading [Schultz94]  Requires large area light source or multiple views Shape from refractive distortion [Murase90]  Limited wave amplitude, orthographic camera model Laser range finders [Wu90]  Specialized equipment

Previous work Shape from refractive irradiance [Jähne92], [Zhang94] & [Daida95]  Requires underwater lens, orthographic camera model

Goals of our system Physically-consistent water surface reconstruction Reconstruction of rapid sequences of flowing, shallow water High reconstruction resolution Use of a minimal number of viewpoints and props

Technical Contributions We present a design for a stereo system for capturing sequences of dynamic water System implementation and results Refractive stereo matching metrics and analysis Effective localization of surface points of shallow water

Refraction Snell’s Law  r 1 sin Θ i = r 2 sin Θ r For air → water: sin Θ i = r w sin Θ r

Imaging water Image point f at q without water Image f at q’ with water qq’ is the refractive disparity

Deriving the surface normal Suppose we know the location of the surface point p and its depth from the camera z We know the angle θ δ between the refracted rays u and v Can compute the incident angle θ i, then the normal n :

Solution space For given refractive disparity, set of solution pairs:  n m z m For every depth z, there is at most one normal n

Reconstruction with Stereo Same setup as with one camera, but with additional calibrated camera We search through the solution space for a particular refractive disparity We use the second camera to determine the error for each instance of n m z m Return best surface point p

Refractive stereo matching Camera 1 Camera 2 Tank Bottom n2n2 n1n1 n

Matching metric Normal collinearity metric  Measure the angle between the two normals n 1 and n 2 to give an error. Disparity difference metric  Swap n 1 and n 2 and reproject to tank plane, measure disparity from the projection before swapping.  Seeks to minimize error due to inaccurate normal measurements as water depth approaches localization error range.

Disparity Difference Metric Camera 1 Camera 2 Tank Bottom e1e1 e2e2

Metric Comparison Disparity difference metric in red Normal collinearity metric in blue

Implementation details Pattern choice  Checkered pattern used Tracking pattern and localization  Lucas-Kanade matching Interpolation of the discrete pattern

System Inputs Calibrated stereo camera system Images of pattern without water from both cameras to give refractive disparities Distorted pattern image sequences

Corner tracking In order to reconstruct a sequence of frames, the corners must be localized at every frame We employ a Lucas-Kanade matching technique, matching templates of the corners to the next frame

Corner Interpolation We cannot assume that our verification ray will land on one of the corners We thus find the four nearest non-collinear corners The surface may be distorted so we cannot assume a grid formation We interpolate between these corners to find the distortion of the verification ray

Results Ripple Drop Waves Pouring water

Future Work Global surface minimization vs local Planar tank constraint removal More complex water scenario capturing

References [Jähne92] B. Jähne, J. Klinke, P. Geissler, and F. Hering. Image sequence analysis of ocean wind waves. In Proc. International Seminar on Imaging in Transport Processes, [Murase90] H. Murase. Shape reconstruction of an undulating transparent object. In Proc. IEEE Intl. Conf. Computer Vision, pages 313–317, [Schultz94] H. Schultz. Retrieving shape information from multiple images of a specular surface. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(2):195–201, [Wu90] Z. Wu and G. A. Meadows. 2-D surface reconstruction of water waves. In Engineering in the Ocean Environment. Conference Proceedings, pages 416–421, [Zhang94] X. Zhang and C. Cox. Measuring the two-dimensional structure of a wavy water surface optically: A surface gradient detector. Experiments in Fluids, Springer Verlag, 17:225–237, 1994.