Photoconsistency constraint C2 q C1 p l = 2 l = 3 Depth labels If this 3D point is visible in both cameras, pixels p and q should have similar intensities.

Slides:



Advertisements
Similar presentations
Pose Estimation and Segmentation of People in 3D Movies Karteek Alahari, Guillaume Seguin, Josef Sivic, Ivan Laptev Inria, Ecole Normale Superieure ICCV.
Advertisements

Graph Cut Algorithms for Computer Vision & Medical Imaging Ramin Zabih Computer Science & Radiology Cornell University Joint work with Y. Boykov, V. Kolmogorov,
The University of Ontario CS 4487/9687 Algorithms for Image Analysis Multi-Label Image Analysis Problems.
1 s-t Graph Cuts for Binary Energy Minimization  Now that we have an energy function, the big question is how do we minimize it? n Exhaustive search is.
Instructor: Mircea Nicolescu Lecture 13 CS 485 / 685 Computer Vision.
Computer Vision Optical Flow
Learning to Detect A Salient Object Reporter: 鄭綱 (3/2)
Last Time Pinhole camera model, projection
Segmentation CSE P 576 Larry Zitnick Many slides courtesy of Steve Seitz.
Multiview stereo. Volumetric stereo Scene Volume V Input Images (Calibrated) Goal: Determine occupancy, “color” of points in V.
Feature matching and tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on.
Multi-view stereo Many slides adapted from S. Seitz.
High-Quality Video View Interpolation
Clustering Color/Intensity
Lecture 11: Structure from motion CS6670: Computer Vision Noah Snavely.
The plan for today Camera matrix
Recovering Articulated Object Models from 3D Range Data Dragomir Anguelov Daphne Koller Hoi-Cheung Pang Praveen Srinivasan Sebastian Thrun Computer Science.
CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry Some material taken from:  David Lowe, UBC  Jiri Matas, CMP Prague
Stereo Computation using Iterative Graph-Cuts
Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.
Matching Compare region of image to region of image. –We talked about this for stereo. –Important for motion. Epipolar constraint unknown. But motion small.
Stereo Matching & Energy Minimization Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski.
Accurate, Dense and Robust Multi-View Stereopsis Yasutaka Furukawa and Jean Ponce Presented by Rahul Garg and Ryan Kaminsky.
Stereo matching Class 10 Read Chapter 7 Tsukuba dataset.
Manhattan-world Stereo Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
Computer vision: models, learning and inference
Image Segmentation Image segmentation is the operation of partitioning an image into a collection of connected sets of pixels. 1. into regions, which usually.
Generating panorama using translational movement model.
A Selective Overview of Graph Cut Energy Minimization Algorithms Ramin Zabih Computer Science Department Cornell University Joint work with Yuri Boykov,
Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee.
What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun.
Joint Depth Map and Color Consistency Estimation for Stereo Images with Different Illuminations and Cameras Yong Seok Heo, Kyoung Mu Lee and Sang Uk Lee.
Graph Cut Algorithms for Binocular Stereo with Occlusions
Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images (Fri) Young Ki Baik, Computer Vision Lab.
Object Stereo- Joint Stereo Matching and Object Segmentation Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on Michael Bleyer Vienna.
Auditory and Visual Spatial Sensing Stan Birchfield Department of Electrical and Computer Engineering Clemson University.
Medical Imaging Dr. Mohammad Dawood Department of Computer Science University of Münster Germany.
Geometry 3: Stereo Reconstruction Introduction to Computer Vision Ronen Basri Weizmann Institute of Science.
Motion Segmentation By Hadas Shahar (and John Y.A.Wang, and Edward H. Adelson, and Wikipedia and YouTube) 1.
Stereo Many slides adapted from Steve Seitz.
CS 4487/6587 Algorithms for Image Analysis
Computer Vision, Robert Pless
CS332 Visual Processing Department of Computer Science Wellesley College Binocular Stereo Vision Region-based stereo matching algorithms Properties of.
1 University of Texas at Austin Machine Learning Group 图像与视频处理 计算机学院 Motion Detection and Estimation.
1 Markov Random Fields with Efficient Approximations Yuri Boykov, Olga Veksler, Ramin Zabih Computer Science Department CORNELL UNIVERSITY.
Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.
Presenter : Kuang-Jui Hsu Date : 2011/3/24(Thur.).
Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image.
1 Motion Analysis using Optical flow CIS601 Longin Jan Latecki Fall 2003 CIS Dept of Temple University.
Gaussian Mixture Models and Expectation-Maximization Algorithm.
Non-Ideal Iris Segmentation Using Graph Cuts
Solving for Stereo Correspondence Many slides drawn from Lana Lazebnik, UIUC.
A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.
Graph Algorithms for Vision Amy Gale November 5, 2002.
Discriminative Training and Machine Learning Approaches Machine Learning Lab, Dept. of CSIE, NCKU Chih-Pin Liao.
Medical Image Analysis Dr. Mohammad Dawood Department of Computer Science University of Münster Germany.
Representing Moving Images with Layers J. Y. Wang and E. H. Adelson MIT Media Lab.
MASKS © 2004 Invitation to 3D vision Lecture 3 Image Primitives andCorrespondence.
Energy functions f(p)  {0,1} : Ising model Can solve fast with graph cuts V( ,  ) = T[  ] : Potts model NP-hard Closely related to Multiway Cut Problem.
Energy minimization Another global approach to improve quality of correspondences Assumption: disparities vary (mostly) smoothly Minimize energy function:
776 Computer Vision Jan-Michael Frahm Spring 2012.
University of Ioannina
Markov Random Fields with Efficient Approximations
Computer Vision Lecture 12: Image Segmentation II
Representing Moving Images with Layers
Multiway Cut for Stereo and Motion with Slanted Surfaces
Representing Moving Images with Layers
Haim Kaplan and Uri Zwick
Image and Video Processing
Image Segmentation.
Presentation transcript:

Photoconsistency constraint C2 q C1 p l = 2 l = 3 Depth labels If this 3D point is visible in both cameras, pixels p and q should have similar intensities

Photoconsistency neighborhood C1 p C2 q l = 2 l = 3 Depth labels Photoconsistency neighbors

Data (photoconsistency) term Photoconsistency neighborhood N photo Arbitrary set of pairs of 3D points (same depth) Current implementation: if the projection of on C2 is nearest to q Our data penalty for configuration f is Note that f p = f q =l by definition of N photo

Data term is regular

Tsukuba images Our results, 4 interactions

Comparison Our results, 10 interactionsBest results [SS ’02]

Expectation-Maximization A powerful technique for many computer vision problems

A simple problem – Line fitting Goal: To group a bunch of points into two “best-fit” line segments

“Chicken-egg problem” If we knew which line each point belonged to, w could compute the best-fit lines.

Chicken-egg problem If we knew what the two best-fit lines were, we could find out which line each point belonged to.

Expectation-Maximization (EM) Initialize: Make random guess for lines Repeat: Find the line closest to each point and group into two sets. (Expectation Step) Find the best-fit lines to the two sets (Maximization Step) Iterate until convergence The algorithm is guaranteed to converge to some local optima

Example:

Converged!

Multiway Cut for Stereo and Motion with Slanted Surfaces Stan Birchfield and Carlo Tomasi ICCV 1999

Motivation Why does it look so bad? an image from a stereo pairdisparity map from graph cuts

Solution Think of this as a segmentation Fit plane to each region to give more accurate results Once you have these planes, reassign pixels to get better fit an image from a stereo pairdisparity map from graph cuts

Algorithm 1. Initialize a set of pixel labels Run graph cuts with integer disparities 2. Fit a plane to each region (connected component) They solve for an affine transformation that best aligns region in left image to corresponding region in right image 3. Assign labels (planes) to pixels Use graph cuts, of course! 4. Repeat Steps 2 & 3 until convergence This style of algorithm should look familiar...

Stereo results

Multimodal Stereo with Graph Cuts Kim, Kolmogorov and Zabih ICCV 2003

Multimodal stereo Suppose the two cameras are different Internal parameters, or modalities There is some consistent mapping of intensities  between them At the right disparity, I 1 (p)   (I 2 (p+d))

Just do it? Problem input has no assignment cost How can we tell how much p likes d ? D(p,d) = (I 1 (p) – µ(I 2 (p+d))) 2 Depends, obviously, on µ We could compute µ from right f Suggests an iterative (EM) approach Alternate between estimating the assignment costs D and the labeling f

Joint intensity histogram Right intensity of corresponding pixel Left intensity

EM-style approach When f is correct, the joint histogram will be highly “concentrated” And vice-versa For a given f, we can construct an assignment cost that tends to make the joint histogram more concentrated Iterative EM-style algorithm Find f given the assignments costs Find the assignment costs given f

Right intensity of corresponding pixel Left intensity Good: Low cost Bad: High cost Assignment costs

Formalizing this Depends on labeling Compute D n+1 from f n

Properties Usable with other matching algorithms Can handle spatially varying µ Need decent D 0, and D n+1  D n For correspondence, easily true (why?) Related to Mutual Information With right formula for D

Results with synthetic distortions

Mutual information (MI) Very powerful method for multimodal registration (not correspondence) Find the affine transformation (warping) of I1 that makes it most similar to I2 A warping implies a joint histogram A disparity map is just a very complex warping MI measures joint histogram “concentration” Search via gradient descent Very successful in practice, and widely used

Use MI for correspondence? No obvious way to apply it A disparity map is a warping with way too many parameters Turn MI into an assignment cost? MI( f ) depends on the joint histogram Each pixel doesn’t independently make the joint histogram concentrated

Relationship with MI Suppose the joint histogram of f is similar to that of g Not equivalent to assuming f similar to g Take the first term of the Taylor expansion for MI( f ) centered at g This is a sum over pixels With the right choice of D, we can approximate MI Choice of D is fairly natural

Summary Multimodal correspondence can be solved using energy minimization Assignment costs depend on labeling Approximation of MI EM-style use of graph cuts Experimental results look promising Not much in the way of guarantees!