The University of Ontario From photohulls to photoflux optimization Yuri Boykov University of Western Ontario Victor Lempitsky Moscow State University.

Slides:



Advertisements
Similar presentations
Using Strong Shape Priors for Multiview Reconstruction Yunda SunPushmeet Kohli Mathieu BrayPhilip HS Torr Department of Computing Oxford Brookes University.
Advertisements

Multi-view Stereo via Volumetric Graph-cuts George Vogiatzis, Philip H. S. Torr Roberto Cipolla.
Multi-view Stereo via Volumetric Graph-cuts
Generating Classic Mosaics with Graph Cuts Y. Liu, O. Veksler and O. Juan University of Western Ontario, Canada Ecole Centrale de Paris, France.
MRI Brain Extraction using a Graph Cut based Active Contour Model Noha Youssry El-Zehiry Noha Youssry El-Zehiry and Adel S. Elmaghraby Computer Engineering.
Level set based Image Segmentation Hang Xiao Jan12, 2013.
Stereo matching Class 7 Read Chapter 7 of tutorial Tsukuba dataset.
Multiview Reconstruction. Why More Than 2 Views? BaselineBaseline – Too short – low accuracy – Too long – matching becomes hard.
Some books on linear algebra Linear Algebra, Serge Lang, 2004 Finite Dimensional Vector Spaces, Paul R. Halmos, 1947 Matrix Computation, Gene H. Golub,
ICCV 2007 tutorial on Discrete Optimization Methods in Computer Vision part I Basic overview of graph cuts.
Corp. Research Princeton, NJ Computing geodesics and minimal surfaces via graph cuts Yuri Boykov, Siemens Research, Princeton, NJ joint work with Vladimir.
Deformable Contours Dr. E. Ribeiro.
Corp. Research Princeton, NJ Cut Metrics and Geometry of Grid Graphs Yuri Boykov, Siemens Research, Princeton, NJ joint work with Vladimir Kolmogorov,
MultiView Stereo Steve Seitz CSE590SS: Vision for Graphics.
Multiview Reconstruction. Why More Than 2 Views? BaselineBaseline – Too short – low accuracy – Too long – matching becomes hard.
Silhouettes in Multiview Stereo Ian Simon. Multiview Stereo Problem Input: – a collection of images of a rigid object (or scene) – camera parameters for.
Last Time Pinhole camera model, projection
Background Removal of Multiview Images by Learning Shape Priors Yu-Pao Tsai, Cheng-Hung Ko, Yi-Ping Hung, and Zen-Chung Shih.
Robust Higher Order Potentials For Enforcing Label Consistency
Project 3 extension: Wednesday at noon Final project proposal extension: Friday at noon >consult with Steve, Rick, and/or Ian now! Project 2 artifact winners...
Lecture 12: Structure from motion CS6670: Computer Vision Noah Snavely.
Computational Photography: Image-based Modeling Jinxiang Chai.
CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai.
EE663 Image Processing Edge Detection 2 Dr. Samir H. Abdul-Jauwad Electrical Engineering Department King Fahd University of Petroleum & Minerals.
Project 2 winners Think about Project 3 Guest lecture on Monday: Aseem Announcements.
The University of Ontario University of Bonn July 2008 Optimization of surface functionals using graph cut algorithms Yuri Boykov presenting joint work.
Today: Image Segmentation Image Segmentation Techniques Snakes Scissors Graph Cuts Mean Shift Wednesday (2/28) Texture analysis and synthesis Multiple.
Multiview stereo. Volumetric stereo Scene Volume V Input Images (Calibrated) Goal: Determine occupancy, “color” of points in V.
Multi-view stereo Many slides adapted from S. Seitz.
The plan for today Camera matrix
Stereo Computation using Iterative Graph-Cuts
Project 1 grades out Announcements. Multiview stereo Readings S. M. Seitz and C. R. Dyer, Photorealistic Scene Reconstruction by Voxel Coloring, International.
Multiview Reconstruction. Why More Than 2 Views? BaselineBaseline – Too short – low accuracy – Too long – matching becomes hard.
Announcements Project Update Extension: due Friday, April 20 Create web page with description, results Present your project in class (~10min/each) on Friday,
Presentation By Michael Tao and Patrick Virtue. Agenda History of the problem Graph cut background Compute graph cut Extensions State of the art Continued.
Lecture 25: Multi-view stereo, continued
Accurate, Dense and Robust Multi-View Stereopsis Yasutaka Furukawa and Jean Ponce Presented by Rahul Garg and Ryan Kaminsky.
Review: Binocular stereo If necessary, rectify the two stereo images to transform epipolar lines into scanlines For each pixel x in the first image Find.
S I E M E N S C O R P O R A T E R E S E A R C H 1 1 Computing Exact Discrete Minimal Surfaces: Extending and Solving the Shortest Path Problem in 3D with.
Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004.
Finish Adaptive Space Carving Anselmo A. Montenegro †, Marcelo Gattass ‡, Paulo Carvalho † and Luiz Velho † †
3D Surface Reconstruction from 2D Images (Survey)
Local invariant features Cordelia Schmid INRIA, Grenoble.
7.1. Mean Shift Segmentation Idea of mean shift:
Graph Cut Algorithms for Binocular Stereo with Occlusions
Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images (Fri) Young Ki Baik, Computer Vision Lab.
The University of Ontario How to fit a surface to a point cloud? or optimization of surface functionals in computer vision Yuri Boykov TRICS seminar Computer.
CS 4487/6587 Algorithms for Image Analysis
Algorithms for MAP estimation in Markov Random Fields Vladimir Kolmogorov University College London.
Local invariant features Cordelia Schmid INRIA, Grenoble.
Finish Hardware Accelerated Voxel Coloring Anselmo A. Montenegro †, Luiz Velho †, Paulo Carvalho † and Marcelo Gattass ‡ †
2D Texture Synthesis Instructor: Yizhou Yu. Texture synthesis Goal: increase texture resolution yet keep local texture variation.
875 Dynamic Scene Reconstruction
Pseudo-Bound Optimization for Binary Energies Meng Tang 1 Ismail Ben Ayed 2 Yuri Boykov 1 1 University of Western Ontario, Canada 2 GE Healthcare Canada.
Active Lighting for Appearance Decomposition Todd Zickler DEAS, Harvard University.
Dynamic Programming (DP), Shortest Paths (SP)
Photoconsistency constraint C2 q C1 p l = 2 l = 3 Depth labels If this 3D point is visible in both cameras, pixels p and q should have similar intensities.
Robot Vision SS 2011 Matthias Rüther / Matthias Straka 1 ROBOT VISION Lesson 7: Volumetric Object Reconstruction Matthias Straka.
A Globally Optimal Algorithm for Robust TV-L 1 Range Image Integration Christopher Zach VRVis Research Center Thomas Pock, Horst Bischof.
The University of Ontario CS 4487/9687 Algorithms for Image Analysis Combining Color and Boundary and other extensions (graph cuts)
Graph Cuts vs. Level Sets
Image gradients and edges
From Photohulls to Photoflux Optimization
Announcements Midterm due now Project 2 artifacts: vote today!
Multi-view reconstruction
Filtering Things to take away from this lecture An image as a function
Discrete Optimization Methods Basic overview of graph cuts
Filtering An image as a function Digital vs. continuous images
IT472 Digital Image Processing
Announcements Project 3 out today (help session at end of class)
Presentation transcript:

The University of Ontario From photohulls to photoflux optimization Yuri Boykov University of Western Ontario Victor Lempitsky Moscow State University

The University of Ontario Overview n Current methods for multi-view reconstruction Photohulls and space carving “Deformable models” n Flux for image segmentation n Photoflux for stereo n Proof of concept

The University of Ontario Kutulakos and Seitz, 2000 Photohull S1S1 S2S2 binary, monotonic photoconsistency local decision to “carve” inconsistent points is OK region growing X it is guaranteed that

The University of Ontario A slide from Kutulakos and Seitz (IJCV 2000) Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene V V

The University of Ontario V Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene ? V

The University of Ontario Photohull may not be well defined in some cases Rectangle of any size is a photoconsistent surface for this camera configuration object of arbitrary shape and surface texture

The University of Ontario Space carving may underestimate the tightest bound on the true scene a larger photoconsistent surface photohull [K&S’00] Why?

The University of Ontario current voxels Photohull surface points Space Carving voxels inconsistent voxel current surface consistent surface

The University of Ontario Non-monotone photoconsistency S1S1 S2S2 non-monotonic photoconsistency NOT safe to “carve” inconsistent points X it is possible that

The University of Ontario Non-binary photoconsistency S1S1 S2S2 non-binary photoconsistency “local” answer no longer available To carve or not to carve? X

The University of Ontario So,… what exactly is photohull ? n Largest continuous photoconsistent surface? Defined only for binary monotone photoconsistency In some cases it may not exist OR n The output of voxel carving algorithm? Which version? –approach to voxel vs. surface consistency –threshold for making 0-1 photoconsistency decisions –order of voxel carving for non-monotone photoconsistency

The University of Ontario Minimal surfaces for multi-view reconstruction X Faugeras and Keriven, 98 Pons et al., 05 Vogiatzis et al., 05 non-binary photoconsistency S1S1 “continuous space carving” ?

The University of Ontario X S1S X NO surface can both contract or expand even for monotonic photoconsistency Is the surface guaranteed to contract? S2S2

The University of Ontario Bottom line n Photohull surface based concept –voxel based algorithms –may not exist binary photoconsistency –thresholding, may leak spatial monotonicity –greedy carving can get fine details but noisy n Deformable models different surface based concept -surface based algorithms -solution exists for solid PDE’s real-valued photoconsistency - energy-based regularization no spatial monotonisity -not greedy, may backtrack/expand no noise but may oversmooth

The University of Ontario Multi-view Reconstruction versus Image Segmentation multi-view reconstruction (volumetric approach) image segmentation Greedy methods Regularization Flux-based methods Thresholding Region growing Voxel coloring Seitz and Dyer, 1997 Space carving Kutulakos and Seitz, 2002 Snakes Kass et al., 1988 Level-sets Malladi et al., 1994 Graph cuts Boykov and Jolly, 2001 Mash-based Esteban and Schmitt, 2004 Level-sets Faugeras and Keriven, 1998 Graph cuts Vogiatzis et al., 2005 Level-sets Vasilevsky and Sidiqqi, 2002 Kimmel et al., 2003 Graph cuts Kolmogorov and Boykov, 2005 This work

The University of Ontario Flux n vector field: some vector defined at each point p “stream of water” with a given speed at each location C1 n flux: “amount of water” passing through a given contour flux(C1) > flux(C2) C2 n n changes sign with orientation

The University of Ontario Why flux? n Regularization: shrinks n Flux: intelligent ballooning flux and length are geometric properties of boundary with opposite effect Gauss-Ostrogradsky (divergence) theorem

The University of Ontario What flux is good for? Segmentation of thin objects [Vasilevsky,Siddiqi’02] boundary with large flux Flux of image gradients:

The University of Ontario boundary with large flux What flux is good for? Segmentation of thin objects [Vasilevsky,Siddiqi’02] Flux of image gradients: Laplacian of image intensities _ _ _ _ _ …or region optimizing Laplacian _ _ + + +

The University of Ontario What flux is good for? Data-driven ballooning Vasilevsky and Sidiqqi, 2002 Kimmel and Bruckstein, 2003 (in the context of level-sets) image intensities on a scan line Laplacian of intensities Kolmogorov and Boykov 2005 (in the context of graph cuts) flux Laplacian zero crossings (if vectors are gradients of some potential field)

The University of Ontario Why flux? Riemannian length + Flux [Kimmel,Bruckstein’03] Riemannian length Flux of Riemannian length + Flux

The University of Ontario Integrating Laplacian Zero-crossings into Graph Cuts [Kolmogorov&Boykov’05] The image is courtesy of David Fleet University of Toronto graph cuts Flux = smart ballooning counteract shrinking bias

The University of Ontario “Shrinking” bias in stereo CVPR’05 slides from Vogiatzis, Torr, Cippola

The University of Ontario Uniform ballooning CVPR’05 slides from Vogiatzis, Torr, Cippola

The University of Ontario Our approach: introduce flux for multi-view stereo n Capture some properties of photohull through a novel surface functional photoflux n Photoflux can be combined with regularization combines benefits of space carving and deformable models –can recover fine details while keeping the noise low n Photoflux addresses shrinking or “over-smoothing” bias of standard regularization methods for N-view reconstruction data-driven intelligent ballooning addresses shrinking bias regularized Laplacian zero crossings for boundary alignment

The University of Ontario X’ From photohull to photoflux for all points X’ right outside photohull X photohull for all points X on photohull binary photoconsistency

The University of Ontario for all points X’ right outside photohull for all points X on photohull X’ From photohull to photoflux photohull X Gradient of photoconsistency is large for all points X on photohull non-binary photoconsistency

The University of Ontario Photoflux X fixed shape S allows to compute global visibility of point X f(S) = outward surface normal at point X

The University of Ontario Photoflux photoflux + regularization n Optimization Local via level-sets or PDE-cuts [BKCD, ECCV’06] maintaining global visibility could be expensive in practice local patch dS can approximate visibility –global graph cuts on a complex [Lempitsky et al. ECCV’06] n other options also… E(S) =

The University of Ontario Photoflux f(S) = visibility vector

The University of Ontario Visibility vector 1 2 X photoconsistency

The University of Ontario Generalization of photoflux idea f(S) = unknown visibility given point X is on some surface if visibility w at point X is given, e.g. w=w(S) patch based visibility w=w(N)

The University of Ontario Photoflux for estimated local photoconsistency gradients Vector field can be computed at every X without fixing any surfce S

The University of Ontario Back to example from K&S’00

The University of Ontario Back to example from K&S’00 gradients and divergence of “photoconsistency” zoom

The University of Ontario Back to example from K&S’00 gradients and divergence of “photoconsistency” photohull [K&S, IJCV’00]

The University of Ontario Back to example from K&S’00 larger photoconsistent surface gradients and divergence of “photoconsistency”

The University of Ontario Back to example from K&S’00 4 cameras 16 cameras

The University of Ontario Textured and non-textured objects 4 cameras 16 cameras

The University of Ontario Regularizing Photoflux photoflux + regularization n Photoflux does not have regularization by itself n Regularized Liplacian zero-crossings E(S) =

The University of Ontario Algorithms n Level-sets n Continuous max-flow techniques n Discrete max-flow (graph cuts) implicit explicit

The University of Ontario “Implicit” graph cuts Most current graph cuts technique implicitly use surfaces represented via binary (interior/exterior) labeling of pixels

The University of Ontario “Explicit” graph cuts Except, a recent explicit surfaces representation method - Kirsanov and Gortler, 2004

The University of Ontario “Explicit” graph cuts n for multi-view reconstruction Lempitsky et al., ECCV 2006 n Explicit surface patches allow local estimation of visibility when computing globally optimal solution Compare with Vogiatzis et al., CVPR’05 approach for visibility local oriented visibility estimate

The University of Ontario “Explicit” graph cuts Regularization + uniform ballooning some details are still over-smoothed Lempitsky et al., ECCV 2006

The University of Ontario Regularization + “intelligent ballooning” Low noise and no shrinking This work

The University of Ontario Space carving

The University of Ontario Also tried surface based flux E(S) = sometimes not submodular

The University of Ontario Photoflux II “Camel”

The University of Ontario Photoflux II “Hand” data courtesy of K. Kutulakos and S Seitz