1. Stereo Matching & Energy Minimization Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski

2. 2/5/2001Vision for Graphics2 Stereo Matching What are some possible algorithms? match “features” and interpolate match edges and interpolate match all pixels with windows (coarse-fine) use optimization: –iterative updating –energy minimization (regularization, stochastic) –dynamic programming –graph algorithms

3. 2/5/2001Vision for Graphics3 Feature-based stereo Match “corner” (interest) points Interpolate complete solution

4. 2/5/2001Vision for Graphics4 Data interpolation Given a sparse set of 3D points, how do we interpolate to a full 3D surface? Scattered data interpolation [Nielson93] triangulate put onto a grid and fill (use pyramid?) place a kernel function over each data point minimize an energy function

5. 2/5/2001Vision for Graphics5 Energy minimization 1-D example: approximating splines z x,y d x,y

6. 2/5/2001Vision for Graphics6 Relaxation Iteratively improve a solution by locally minimizing the energy: relax to solution Earliest application: WWII numerical simulations z x,y d x,y d x+1,y

7. 2/5/2001Vision for Graphics7 Relaxation Try solving this problem yourself: 1.Make up a bunch of z x,y 2.Guess best values for d x,y

8. 2/5/2001Vision for Graphics8 Relaxation How can we get the best solution? Differentiate energy function, set to 0

9. 2/5/2001Vision for Graphics9 Non-quadratic energy How about minimizing this cost function?

10. 2/5/2001Vision for Graphics10 Discrete optimization space What if you have discrete (e.g., binary) values? 00 11 0 1 0 1 0 1

11. 2/5/2001Vision for Graphics11 Dynamic programming Evaluate best cumulative cost at each pixel 00 11 0 1 0 1 0 1

12. 2/5/2001Vision for Graphics12 Dynamic programming 1-D cost function

13. 2/5/2001Vision for Graphics13 Dynamic programming Can we apply this trick in 2D as well? d x,y d x-1,y d x,y-1 d x-1,y-1 No: d x,y-1 and d x-1,y may depend on different values of d x-1,y-1

14. 2/5/2001Vision for Graphics14 Graph cuts Solution technique for general 2D problem

15. 2/5/2001Vision for Graphics15 Graph cuts Two different kinds of moves:

16. 2/5/2001Vision for Graphics16 Graph cuts  -  swap: interchange  and  labels

17. 2/5/2001Vision for Graphics17 Graph cuts  expansion: add pixels to  class

18. Back to stereo matching…

19. 2/5/2001Vision for Graphics19 Neighborhood size (review) Smaller neighborhood: more details Larger neighborhood: fewer isolated mistakes w = 3w = 20

20. 2/5/2001Vision for Graphics20 Plane sweep stereo Re-order (pixel / disparity) evaluation loops for every pixel,for every disparity for every disparity for every pixel compute cost compute cost

21. 2/5/2001Vision for Graphics21 Stereo matching framework 1.For every disparity, compute raw matching costs Why use a robust function? occlusions, other outliers Can also use alternative match criteria

22. 2/5/2001Vision for Graphics22 Stereo matching framework 2.Aggregate costs spatially Here, we are using a box filter (efficient moving average implementation) Can also use weighted average, [non-linear] diffusion…

23. 2/5/2001Vision for Graphics23 Stereo matching framework 3.Choose winning disparity at each pixel Can interpolate to sub-pixel accuracy

24. 2/5/2001Vision for Graphics24 Linear diffusion Average energy with neighbors windowdiffusion

25. 2/5/2001Vision for Graphics25 Linear diffusion Average energy with neighbors + starting value windowdiffusion

26. 2/5/2001Vision for Graphics26 Dynamic programming 1-D cost function [Intille & Bobick, IJCV 99]

27. 2/5/2001Vision for Graphics27 Dynamic programming Disparity space image and min. cost path

28. 2/5/2001Vision for Graphics28 Dynamic programming Sample result (note horizontal streaks)

29. 2/5/2001Vision for Graphics29 Graph cuts  -  swap  expansion modify smoothness penalty based on edges compute best possible match within integer disparity

30. 2/5/2001Vision for Graphics30 Bayesian inference Formulate as statistical inference problem Prior model p P (d) Measurement modelp M (I L, I R | d) Posterior model p M (d | I L, I R )  p P (d) p M (I L, I R | d) Maximum a Posteriori (MAP estimate): maximize p M (d | I L, I R )

31. 2/5/2001Vision for Graphics31 Markov Random Field Probability distribution on disparity field d(x,y) Enforces smoothness or coherence on field

32. 2/5/2001Vision for Graphics32 Measurement model Likelihood of intensity correspondence Corresponds to Gaussian noise for quadratic 

33. 2/5/2001Vision for Graphics33 MAP estimate Maximize posterior likelihood Equivalent to regularization (energy minimization with smoothness constraints)

34. 2/5/2001Vision for Graphics34 Why Bayesian estimation? Principled way of determining cost function Explicit model of noise and prior knowledge Admits a wider variety of optimization algorithms: gradient descent (local minimization) stochastic optimization (Gibbs Sampler) mean-field optimization graph theoretic (actually deterministic) [Zabih]

35. 2/5/2001Vision for Graphics35 Mean-field interpretation Bayesian non-linear diffusion rule: update your probability distribution assuming your neighbors’ distributions are independent (valid for Markov chain) Equivalent to finding best factored approximation P(d|I L,I R ) ~ Q(d) =  i Q i (d i )

36. 2/5/2001Vision for Graphics36 Mean-field interpretation log MAP estimate -log P(d|I L,I R ) =  ij E ij (d i,d j ) +  i E i (d i ) =  ij s i A ij s i +  i b i s i Kullback-Leibler divergence D KL =H(Q) -  d Q(d) log P(d) =  ik q ik log q ik +  ij s i A ij s i +  i b i s i

37. 2/5/2001Vision for Graphics37 Mean-field interpretation minimize K-L divergence with  k q ik = 1 update rule: q ik  exp[ - (  j a ij k q j + b ik )] = exp[- (  jl E ij (d i =k,d j =l)p(d j =l) + E i (d i =k) )]

38. 2/5/2001Vision for Graphics38 Depth Map Results Input imageSum Abs Diff Mean fieldGraph cuts

39. 2/5/2001Vision for Graphics39 Stereo with Non-Linear Diffusion Advantages: works very well in non-occluded regions Disadvantages: restricted to two images (not) gets confused in occluded regions can’t handle mixed pixels

40. 2/5/2001Vision for Graphics40 Summary Applications Image rectification Matching criteria Local algorithms (aggregation) area-based; iterative updating Optimization algorithms: energy (cost) formulation Markov Random Fields mean-field; dynamic programming; stochastic; graph algorithms

41. 2/5/2001Vision for Graphics41 More stereo…(next 2 lectures) Multi-image stereo Volumetric techniques Graph cuts Transparency Surfaces and level sets

42. 2/5/2001Vision for Graphics42 Bibliography See the references in the readings… Y. Boykov, O. Veksler, and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, Unpublished manuscript, 2000. A.F. Bobick and S.S. Intille. Large occlusion stereo. International Journal of Computer Vision, 33(3), September 1999. pp. 181-200 D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998 R. Szeliski. Stereo algorithms and representations for image-based rendering. In British Machine Vision Conference (BMVC'99), volume 2, pages 314-328, Nottingham, England, September 1999. R. Szeliski and R. Zabih. An experimental comparison of stereo algorithms. In International Workshop on Vision Algorithms, pages 1- 19, Kerkyra, Greece, September 1999. G. M. Nielson, Scattered Data Modeling, IEEE Computer Graphics and Applications, 13(1), January 1993, pp. 60-70.