The University of Ontario From photohulls to photoflux optimization Yuri Boykov University of Western Ontario Victor Lempitsky Moscow State University

The University of Ontario Overview n Current methods for multi-view reconstruction Photohulls and space carving “Deformable models” n Flux for image segmentation n Photoflux for stereo n Proof of concept

The University of Ontario Kutulakos and Seitz, 2000 Photohull S1S1 S2S2 binary, monotonic photoconsistency local decision to “carve” inconsistent points is OK region growing X it is guaranteed that

The University of Ontario A slide from Kutulakos and Seitz (IJCV 2000) Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene V V

The University of Ontario V Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene ? V

The University of Ontario Photohull may not be well defined in some cases Rectangle of any size is a photoconsistent surface for this camera configuration object of arbitrary shape and surface texture

The University of Ontario Space carving may underestimate the tightest bound on the true scene a larger photoconsistent surface photohull [K&S’00] Why?

The University of Ontario current voxels Photohull surface points Space Carving voxels inconsistent voxel current surface consistent surface

The University of Ontario Non-monotone photoconsistency S1S1 S2S2 non-monotonic photoconsistency NOT safe to “carve” inconsistent points X it is possible that

The University of Ontario Non-binary photoconsistency S1S1 S2S2 non-binary photoconsistency “local” answer no longer available To carve or not to carve? X

The University of Ontario So,… what exactly is photohull ? n Largest continuous photoconsistent surface? Defined only for binary monotone photoconsistency In some cases it may not exist OR n The output of voxel carving algorithm? Which version? –approach to voxel vs. surface consistency –threshold for making 0-1 photoconsistency decisions –order of voxel carving for non-monotone photoconsistency

The University of Ontario Minimal surfaces for multi-view reconstruction X Faugeras and Keriven, 98 Pons et al., 05 Vogiatzis et al., 05 non-binary photoconsistency S1S1 “continuous space carving” ?

The University of Ontario X S1S X NO surface can both contract or expand even for monotonic photoconsistency Is the surface guaranteed to contract? S2S2

The University of Ontario Bottom line n Photohull surface based concept –voxel based algorithms –may not exist binary photoconsistency –thresholding, may leak spatial monotonicity –greedy carving can get fine details but noisy n Deformable models different surface based concept -surface based algorithms -solution exists for solid PDE’s real-valued photoconsistency - energy-based regularization no spatial monotonisity -not greedy, may backtrack/expand no noise but may oversmooth

The University of Ontario Multi-view Reconstruction versus Image Segmentation multi-view reconstruction (volumetric approach) image segmentation Greedy methods Regularization Flux-based methods Thresholding Region growing Voxel coloring Seitz and Dyer, 1997 Space carving Kutulakos and Seitz, 2002 Snakes Kass et al., 1988 Level-sets Malladi et al., 1994 Graph cuts Boykov and Jolly, 2001 Mash-based Esteban and Schmitt, 2004 Level-sets Faugeras and Keriven, 1998 Graph cuts Vogiatzis et al., 2005 Level-sets Vasilevsky and Sidiqqi, 2002 Kimmel et al., 2003 Graph cuts Kolmogorov and Boykov, 2005 This work

The University of Ontario Flux n vector field: some vector defined at each point p “stream of water” with a given speed at each location C1 n flux: “amount of water” passing through a given contour flux(C1) > flux(C2) C2 n n changes sign with orientation

The University of Ontario Why flux? n Regularization: shrinks n Flux: intelligent ballooning flux and length are geometric properties of boundary with opposite effect Gauss-Ostrogradsky (divergence) theorem

The University of Ontario What flux is good for? Segmentation of thin objects [Vasilevsky,Siddiqi’02] boundary with large flux Flux of image gradients:

The University of Ontario boundary with large flux What flux is good for? Segmentation of thin objects [Vasilevsky,Siddiqi’02] Flux of image gradients: Laplacian of image intensities _ _ _ _ _ …or region optimizing Laplacian _ _ + + +

The University of Ontario What flux is good for? Data-driven ballooning Vasilevsky and Sidiqqi, 2002 Kimmel and Bruckstein, 2003 (in the context of level-sets) image intensities on a scan line Laplacian of intensities Kolmogorov and Boykov 2005 (in the context of graph cuts) flux Laplacian zero crossings (if vectors are gradients of some potential field)

The University of Ontario Why flux? Riemannian length + Flux [Kimmel,Bruckstein’03] Riemannian length Flux of Riemannian length + Flux

The University of Ontario Integrating Laplacian Zero-crossings into Graph Cuts [Kolmogorov&Boykov’05] The image is courtesy of David Fleet University of Toronto graph cuts Flux = smart ballooning counteract shrinking bias

The University of Ontario “Shrinking” bias in stereo CVPR’05 slides from Vogiatzis, Torr, Cippola

The University of Ontario Uniform ballooning CVPR’05 slides from Vogiatzis, Torr, Cippola

The University of Ontario Our approach: introduce flux for multi-view stereo n Capture some properties of photohull through a novel surface functional photoflux n Photoflux can be combined with regularization combines benefits of space carving and deformable models –can recover fine details while keeping the noise low n Photoflux addresses shrinking or “over-smoothing” bias of standard regularization methods for N-view reconstruction data-driven intelligent ballooning addresses shrinking bias regularized Laplacian zero crossings for boundary alignment

The University of Ontario X’ From photohull to photoflux for all points X’ right outside photohull X photohull for all points X on photohull binary photoconsistency

The University of Ontario for all points X’ right outside photohull for all points X on photohull X’ From photohull to photoflux photohull X Gradient of photoconsistency is large for all points X on photohull non-binary photoconsistency

The University of Ontario Photoflux X fixed shape S allows to compute global visibility of point X f(S) = outward surface normal at point X

The University of Ontario Photoflux photoflux + regularization n Optimization Local via level-sets or PDE-cuts [BKCD, ECCV’06] maintaining global visibility could be expensive in practice local patch dS can approximate visibility –global graph cuts on a complex [Lempitsky et al. ECCV’06] n other options also… E(S) =

The University of Ontario Photoflux f(S) = visibility vector

The University of Ontario Visibility vector 1 2 X photoconsistency

The University of Ontario Generalization of photoflux idea f(S) = unknown visibility given point X is on some surface if visibility w at point X is given, e.g. w=w(S) patch based visibility w=w(N)

The University of Ontario Photoflux for estimated local photoconsistency gradients Vector field can be computed at every X without fixing any surfce S

The University of Ontario Back to example from K&S’00

The University of Ontario Back to example from K&S’00 gradients and divergence of “photoconsistency” zoom

The University of Ontario Back to example from K&S’00 gradients and divergence of “photoconsistency” photohull [K&S, IJCV’00]

The University of Ontario Back to example from K&S’00 larger photoconsistent surface gradients and divergence of “photoconsistency”

The University of Ontario Back to example from K&S’00 4 cameras 16 cameras

The University of Ontario Textured and non-textured objects 4 cameras 16 cameras

The University of Ontario Regularizing Photoflux photoflux + regularization n Photoflux does not have regularization by itself n Regularized Liplacian zero-crossings E(S) =

The University of Ontario Algorithms n Level-sets n Continuous max-flow techniques n Discrete max-flow (graph cuts) implicit explicit

The University of Ontario “Implicit” graph cuts Most current graph cuts technique implicitly use surfaces represented via binary (interior/exterior) labeling of pixels

The University of Ontario “Explicit” graph cuts Except, a recent explicit surfaces representation method - Kirsanov and Gortler, 2004

The University of Ontario “Explicit” graph cuts n for multi-view reconstruction Lempitsky et al., ECCV 2006 n Explicit surface patches allow local estimation of visibility when computing globally optimal solution Compare with Vogiatzis et al., CVPR’05 approach for visibility local oriented visibility estimate

The University of Ontario “Explicit” graph cuts Regularization + uniform ballooning some details are still over-smoothed Lempitsky et al., ECCV 2006

The University of Ontario Regularization + “intelligent ballooning” Low noise and no shrinking This work

The University of Ontario Space carving

The University of Ontario Also tried surface based flux E(S) = sometimes not submodular

The University of Ontario Photoflux II “Camel”

The University of Ontario Photoflux II “Hand” data courtesy of K. Kutulakos and S Seitz