Computer Vision CSE576, Spring 2005 Richard Szeliski

Slides:



Advertisements
Similar presentations
CSE473/573 – Stereo and Multiple View Geometry
Advertisements

875: Recent Advances in Geometric Computer Vision & Recognition
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923.
Gratuitous Picture US Naval Artillery Rangefinder from World War I (1918)!!
Stereo Many slides adapted from Steve Seitz. Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image Where does the.
Recap from Previous Lecture Tone Mapping – Preserve local contrast or detail at the expense of large scale contrast. – Changing the brightness within.
Stereo.
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2012.
MultiView Stereo Steve Seitz CSE590SS: Vision for Graphics.
Last Time Pinhole camera model, projection
Computer Vision CSE576, Spring 2005 Richard Szeliski
CS6670: Computer Vision Noah Snavely Lecture 17: Stereo
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2005 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
Project 2 winners Think about Project 3 Guest lecture on Monday: Aseem Announcements.
Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.
Stereo Matching Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski.
Multi-view stereo Many slides adapted from S. Seitz.
Stereo Binocular Stereo Calibration (finish up) Next Time Motivation
Stereopsis Mark Twain at Pool Table", no date, UCR Museum of Photography.
The plan for today Camera matrix
CS 223b 1 More on stereo and correspondence. CS 223b 2 =?f g Mostpopular For each window, match to closest window on epipolar line in other image. (slides.
3D from multiple views : Rendering and Image Processing Alexei Efros …with a lot of slides stolen from Steve Seitz and Jianbo Shi.
Project 1 grades out Announcements. Multiview stereo Readings S. M. Seitz and C. R. Dyer, Photorealistic Scene Reconstruction by Voxel Coloring, International.
Announcements Project Update Extension: due Friday, April 20 Create web page with description, results Present your project in class (~10min/each) on Friday,
Stereo Matching Computer Vision CSE576, Spring 2005 Richard Szeliski.
CSE473/573 – Stereo Correspondence
Announcements PS3 Due Thursday PS4 Available today, due 4/17. Quiz 2 4/24.
Stereo Guest Lecture by Li Zhang
Project 1 artifact winners Project 2 questions Project 2 extra signup slots –Can take a second slot if you’d like Announcements.
Stereo Matching & Energy Minimization Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski.
Midterm went out on Tuesday (due next Tuesday) Project 3 out today Announcements.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
Accurate, Dense and Robust Multi-View Stereopsis Yasutaka Furukawa and Jean Ponce Presented by Rahul Garg and Ryan Kaminsky.
Stereo matching Class 10 Read Chapter 7 Tsukuba dataset.
CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai.
What have we leaned so far? Camera structure Eye structure Project 1: High Dynamic Range Imaging.
Computer Vision Spring ,-685 Instructor: S. Narasimhan WH 5409 T-R 10:30am – 11:50am Lecture #15.
Camera Calibration & Stereo Reconstruction Jinxiang Chai.
What have we leaned so far? Camera structure Eye structure Project 1: High Dynamic Range Imaging.
Announcements Project 1 artifact winners Project 2 questions
Structure from images. Calibration Review: Pinhole Camera.
Recap from Monday Image Warping – Coordinate transforms – Linear transforms expressed in matrix form – Inverse transforms useful when synthesizing images.
Stereo Readings Szeliski, Chapter 11 (through 11.5) Single image stereogram, by Niklas EenNiklas Een.
Stereo Many slides adapted from Steve Seitz.
Project 2 code & artifact due Friday Midterm out tomorrow (check your ), due next Fri Announcements TexPoint fonts used in EMF. Read the TexPoint.
Stereo Many slides adapted from Steve Seitz. Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image image 1image.
Computer Vision, Robert Pless
Computer Vision Lecture #10 Hossam Abdelmunim 1 & Aly A. Farag 2 1 Computer & Systems Engineering Department, Ain Shams University, Cairo, Egypt 2 Electerical.
Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform  Location of the impulse.
Lecture 16: Stereo CS4670 / 5670: Computer Vision Noah Snavely Single image stereogram, by Niklas EenNiklas Een.
Digital Image Processing
776 Computer Vision Jan-Michael Frahm Spring 2012.
Solving for Stereo Correspondence Many slides drawn from Lana Lazebnik, UIUC.
A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.
Project 2 due today Project 3 out today Announcements TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAA.
Project 3 code & artifact due Tuesday Final project proposals due noon Wed (by ) –One-page writeup (from project web page), specifying: »Your team.
Energy minimization Another global approach to improve quality of correspondences Assumption: disparities vary (mostly) smoothly Minimize energy function:
CSE 185 Introduction to Computer Vision Stereo 2.
Stereo CS4670 / 5670: Computer Vision Noah Snavely Single image stereogram, by Niklas EenNiklas Een.
Noah Snavely, Zhengqi Li
Lecture 11 Homogeneous coordinates. Stereo.
제 5 장 스테레오.
CS4670 / 5670: Computer Vision Kavita Bala Lec 27: Stereo.
STEREOPSIS The Stereopsis Problem: Fusion and Reconstruction
Jeremy Bolton, PhD Assistant Teaching Professor
Announcements Midterms graded (handed back at end of lecture)
What have we learned so far?
Announcements Midterm due now Project 2 artifacts: vote today!
Announcements Project 3 out today (help session at end of class)
Stereo vision Many slides adapted from Steve Seitz.
Presentation transcript:

Computer Vision CSE576, Spring 2005 Richard Szeliski Stereo Matching Computer Vision CSE576, Spring 2005 Richard Szeliski

Stereo Matching Given two or more images of the same scene or object, compute a representation of its shape What are some possible applications? ? CSE 576, Spring 2008 Stereo matching

Face modeling From one stereo pair to a 3D head model [Frederic Deverney, INRIA] CSE 576, Spring 2008 Stereo matching

Z-keying: mix live and synthetic Takeo Kanade, CMU (Stereo Machine) CSE 576, Spring 2008 Stereo matching

input depth image novel view [Matthies,Szeliski,Kanade’88] View Interpolation Given two images with correspondences, morph (warp and cross-dissolve) between them [Chen & Williams, SIGGRAPH’93] input depth image novel view [Matthies,Szeliski,Kanade’88] CSE 576, Spring 2008 Stereo matching

More view interpolation Spline-based depth map input depth image novel view [Szeliski & Kang ‘95] CSE 576, Spring 2008 Stereo matching

View Morphing Morph between pair of images using epipolar geometry [Seitz & Dyer, SIGGRAPH’96] CSE 576, Spring 2008 Stereo matching

Video view interpolation CSE 576, Spring 2008 Stereo matching

CSE 576, Spring 2008 Stereo matching

Virtualized RealityTM [Takeo Kanade et al., CMU] collect video from 50+ stream reconstruct 3D model sequences steerable version used for SuperBowl XXV “eye vision” http://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html CSE 576, Spring 2008 Stereo matching

Real-time stereo Used for robot navigation (and other tasks) Nomad robot searches for meteorites in Antartica http://www.frc.ri.cmu.edu/projects/meteorobot/index.html Used for robot navigation (and other tasks) Software-based real-time stereo techniques CSE 576, Spring 2008 Stereo matching

Additional applications Real-time people tracking (systems from Pt. Gray Research and SRI) “Gaze” correction for video conferencing [Ott,Lewis,Cox InterChi’93] Other ideas? CSE 576, Spring 2008 Stereo matching

Stereo Matching Given two or more images of the same scene or object, compute a representation of its shape What are some possible representations? depth maps volumetric models 3D surface models planar (or offset) layers CSE 576, Spring 2008 Stereo matching

Stereo Matching What are some possible algorithms? match “features” and interpolate match edges and interpolate match all pixels with windows (coarse-fine) use optimization: iterative updating dynamic programming energy minimization (regularization, stochastic) graph algorithms CSE 576, Spring 2008 Stereo matching

Outline (remainder of lecture) Image rectification Matching criteria Local algorithms (aggregation) iterative updating Optimization algorithms: energy (cost) formulation & Markov Random Fields mean-field, stochastic, and graph algorithms Multi-View stereo & occlusions CSE 576, Spring 2008 Stereo matching

Stereo: epipolar geometry Match features along epipolar lines epipolar plane viewing ray epipolar line CSE 576, Spring 2008 Stereo matching

Stereo image pair CSE 576, Spring 2008 Stereo matching

Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923 Anaglyphs http://www.rainbowsymphony.com/freestuff.html (Wikipedia for images) Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923 CSE 576, Spring 2008 Stereo matching

Stereo: epipolar geometry for two images (or images with collinear camera centers), can find epipolar lines epipolar lines are the projection of the pencil of planes passing through the centers Rectification: warping the input images (perspective transformation) so that epipolar lines are horizontal CSE 576, Spring 2008 Stereo matching

Rectification Project each image onto same plane, which is parallel to the epipole Resample lines (and shear/stretch) to place lines in correspondence, and minimize distortion [Loop and Zhang, CVPR’99] CSE 576, Spring 2008 Stereo matching

Rectification BAD! CSE 576, Spring 2008 Stereo matching

Rectification GOOD! CSE 576, Spring 2008 Stereo matching

Finding correspondences apply feature matching criterion (e.g., correlation or Lucas-Kanade) at all pixels simultaneously search only over epipolar lines (many fewer candidate positions) CSE 576, Spring 2008 Stereo matching

Your basic stereo algorithm Improvement: match windows This should look familar... For each pixel in the left image For each epipolar line compare with every pixel on same epipolar line in right image pick pixel with minimum match cost CSE 576, Spring 2008 Stereo matching

Image registration (revisited) How do we determine correspondences? block matching or SSD (sum squared differences) d is the disparity (horizontal motion) How big should the neighborhood be? The key to image mosaicing is how to register two images. And the simplest case for image registration is block matching, or estimating the translational motion between two images. Suppose a template (for example, the green box) or even the whole image moves with the same translation, we can estimate the motion by minimize the squared difference between two images. For instance, we can search the minimum squared error by shifting the template window around its neighborhood. CSE 576, Spring 2008 Stereo matching

Neighborhood size Smaller neighborhood: more details Larger neighborhood: fewer isolated mistakes w = 3 w = 20 CSE 576, Spring 2008 Stereo matching

Matching criteria Raw pixel values (correlation) Band-pass filtered images [Jones & Malik 92] “Corner” like features [Zhang, …] Edges [many people…] Gradients [Seitz 89; Scharstein 94] Rank statistics [Zabih & Woodfill 94] CSE 576, Spring 2008 Stereo matching

Stereo: certainty modeling Compute certainty map from correlations input depth map certainty map CSE 576, Spring 2008 Stereo matching

Plane Sweep Stereo Sweep family of planes through volume  projective re-sampling of (X,Y,Z) virtual camera composite input image each plane defines an image  composite homography CSE 576, Spring 2008 Stereo matching

Plane Sweep Stereo For each depth plane compute composite (mosaic) image — mean compute error image — variance convert to confidence and aggregate spatially Select winning depth at each pixel CSE 576, Spring 2008 Stereo matching

Plane sweep stereo Re-order (pixel / disparity) evaluation loops for every pixel, for every disparity for every disparity for every pixel compute cost compute cost CSE 576, Spring 2008 Stereo matching

Stereo matching framework For every disparity, compute raw matching costs Why use a robust function? occlusions, other outliers Can also use alternative match criteria CSE 576, Spring 2008 Stereo matching

Stereo matching framework Aggregate costs spatially Here, we are using a box filter (efficient moving average implementation) Can also use weighted average, [non-linear] diffusion… CSE 576, Spring 2008 Stereo matching

Stereo matching framework Choose winning disparity at each pixel Interpolate to sub-pixel accuracy E(d) d* d CSE 576, Spring 2008 Stereo matching

Traditional Stereo Matching Advantages: gives detailed surface estimates fast algorithms based on moving averages sub-pixel disparity estimates and confidence Limitations: narrow baseline  noisy estimates fails in textureless areas gets confused near occlusion boundaries CSE 576, Spring 2008 Stereo matching

Feature-based stereo Match “corner” (interest) points Interpolate complete solution CSE 576, Spring 2008 Stereo matching

Data interpolation Given a sparse set of 3D points, how do we interpolate to a full 3D surface? Scattered data interpolation [Nielson93] triangulate put onto a grid and fill (use pyramid?) place a kernel function over each data point minimize an energy function CSE 576, Spring 2008 Stereo matching

Energy minimization 1-D example: approximating splines dx,y zx,y CSE 576, Spring 2008 Stereo matching

Relaxation How can we get the best solution? Differentiate energy function, set to 0 CSE 576, Spring 2008 Stereo matching

Relaxation Iteratively improve a solution by locally minimizing the energy: relax to solution Earliest application: WWII numerical simulations zx,y dx+1,y dx,y dx+1,y CSE 576, Spring 2008 Stereo matching

Dynamic programming Evaluate best cumulative cost at each pixel 1 1 1 1 1 1 1 1 CSE 576, Spring 2008 Stereo matching

Dynamic programming 1-D cost function CSE 576, Spring 2008 Stereo matching

Dynamic programming Disparity space image and min. cost path CSE 576, Spring 2008 Stereo matching

Dynamic programming Sample result (note horizontal streaks) [Intille & Bobick] CSE 576, Spring 2008 Stereo matching

Dynamic programming Can we apply this trick in 2D as well? dx-1,y dx,y No: dx,y-1 and dx-1,y may depend on different values of dx-1,y-1 CSE 576, Spring 2008 Stereo matching

Graph cuts Solution technique for general 2D problem CSE 576, Spring 2008 Stereo matching

Graph cuts a-b swap expansion modify smoothness penalty based on edges compute best possible match within integer disparity CSE 576, Spring 2008 Stereo matching

Graph cuts Two different kinds of moves: CSE 576, Spring 2008 Stereo matching

pM(d | IL, IR)  pP(d) pM(IL, IR| d) Bayesian inference Formulate as statistical inference problem Prior model pP(d) Measurement model pM(IL, IR| d) Posterior model pM(d | IL, IR)  pP(d) pM(IL, IR| d) Maximum a Posteriori (MAP estimate): maximize pM(d | IL, IR) CSE 576, Spring 2008 Stereo matching

Markov Random Field Probability distribution on disparity field d(x,y) Enforces smoothness or coherence on field CSE 576, Spring 2008 Stereo matching

Measurement model Likelihood of intensity correspondence Corresponds to Gaussian noise for quadratic r CSE 576, Spring 2008 Stereo matching

MAP estimate Maximize posterior likelihood Equivalent to regularization (energy minimization with smoothness constraints) CSE 576, Spring 2008 Stereo matching

Why Bayesian estimation? Principled way of determining cost function Explicit model of noise and prior knowledge Admits a wider variety of optimization algorithms: gradient descent (local minimization) stochastic optimization (Gibbs Sampler) mean-field optimization graph theoretic (actually deterministic) [Zabih] [loopy] belief propagation large stochastic flips [Swendsen-Wang] CSE 576, Spring 2008 Stereo matching

Input image Sum Abs Diff Depth Map Results Input image Sum Abs Diff Mean field Graph cuts CSE 576, Spring 2008 Stereo matching

Stereo evaluation CSE 576, Spring 2008 Stereo matching

Stereo—best algorithms CSE 576, Spring 2008 Stereo matching

Traditional stereo Advantages: Disadvantages: works very well in non-occluded regions Disadvantages: restricted to two images (not) gets confused in occluded regions can’t handle mixed pixels CSE 576, Spring 2008 Stereo matching

Multi-View Stereo

Stereo Reconstruction f u u’ baseline z C C’ X Steps Calibrate cameras Rectify images Compute disparity Estimate depth CSE 576, Spring 2008 Stereo matching

Choosing the Baseline What’s the optimal baseline? Large Baseline Small Baseline What’s the optimal baseline? Too small: large depth error Too large: difficult search problem CSE 576, Spring 2008 Stereo matching

Effect of Baseline on Estimation CSE 576, Spring 2008 Stereo matching

CSE 576, Spring 2008 Stereo matching

Multibaseline Stereo Basic Approach Limitations Choose a reference view Use your favorite stereo algorithm BUT replace two-view SSD with SSD over all baselines Limitations Must choose a reference view Visibility: select which frames to match [Kang, Szeliski, Chai, CVPR’01] CSE 576, Spring 2008 Stereo matching

Epipolar-Plane Images [Bolles 87] http://www.graphics.lcs.mit.edu/~aisaksen/projects/drlf/epi/ Lesson: Beware of occlusions CSE 576, Spring 2008 Stereo matching

Active stereo with structured light Li Zhang’s one-shot stereo camera 2 camera 1 projector camera 1 projector Project “structured” light patterns onto the object simplifies the correspondence problem CSE 576, Spring 2008 Stereo matching

Spacetime Stereo Li Zhang, Noah Snavely, Brian Curless, Steve Seitz http://grail.cs.washington.edu/projects/stfaces/ CSE 576, Spring 2008 Stereo matching

Summary Applications Image rectification Matching criteria Local algorithms (aggregation & diffusion) Optimization algorithms energy (cost) formulation & Markov Random Fields mean-field; dynamic programming; stochastic; graph algorithms Multi-View stereo visibility, occlusion-ordered sweeps CSE 576, Spring 2008 Stereo matching

Bibliography D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1):7-42, May 2002. R. Szeliski. Stereo algorithms and representations for image-based rendering. In British Machine Vision Conference (BMVC'99), volume 2, pages 314-328, Nottingham, England, September 1999. G. M. Nielson, Scattered Data Modeling, IEEE Computer Graphics and Applications, 13(1), January 1993, pp. 60-70. S. B. Kang, R. Szeliski, and J. Chai. Handling occlusions in dense multi-view stereo. In CVPR'2001, vol. I, pages 103-110, December 2001. Y. Boykov, O. Veksler, and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, Unpublished manuscript, 2000. A.F. Bobick and S.S. Intille. Large occlusion stereo. International Journal of Computer Vision, 33(3), September 1999. pp. 181-200 D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998 CSE 576, Spring 2008 Stereo matching

Bibliography Volume Intersection Voxel Coloring and Space Carving Martin & Aggarwal, “Volumetric description of objects from multiple views”, Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158. Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32. Voxel Coloring and Space Carving Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc. Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073. Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, pp. 17-24. Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp. 307-314. CSE 576, Spring 2008 Stereo matching

Bibliography Related References Bolles, Baker, and Marimont, “Epipolar-Plane Image Analysis: An Approach to Determining Structure from Motion”, International Journal of Computer Vision, vol 1, no 1, 1987, pp. 7-55. Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344. Szeliski & Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, 517-524. Roy & Cox, “A Maximum-Flow Formulation of the N-camera Stereo Correspondence Problem”, Proc. ICCV, 1998, pp. 492-499. Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-image stereo and shading", International Journal of Computer Vision, 16, 1995, pp. 35-56. Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3-10. CSE 576, Spring 2008 Stereo matching