1. Object Stereo- Joint Stereo Matching and Object Segmentation Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on Michael Bleyer Vienna University of Technology, Austria Carsten Rother Microsoft Research Cambridge, UK Pushmeet Kohli Microsoft Research Cambridge, UK Daniel Scharstein Middlebury College, USA Sudipta Sinha Microsoft Research Redmond, USA 1

2. Outline Introduction Proposed Model Energy Minimization Result Conclusion 2

3. Introduction A 3D scene is represented as a collection of visually distinct and spatially coherent objects. Each object is characterized by three different aspects: color model 3D plane 3D connectivity 3

4. Introduction The proposed method employs object-level color models as a soft constraint to aid depth estimation. The proposed method can recover the depth of regions that are fully occluded in one input view. 4

5. Introduction The proposed method models a 3D scene as a collection of 3D objects, assume that 1.each object is compact. 2.each object is connected. 3.all visible parts of an object share a similar appearance. 4.scene interpretations with a few large objects. 5

6. Introduction Compactness objects are coherent. depth variations within an object are smooth. objects have a bias towards being planar in 3D. 6

7. Introduction 3D Connectivity disconnected 2D regions and separated by smaller depth. 7

8. Introduction Similar Appearance use color as the only appearance cue. each object in a scene has a compact distribution of colors. Scene Interpretation with few objects. prevent single pixels from being explained as individual objects. 8

9. Introduction Color models introduce a color segmentation into the stereo matching process. assign untextured regions to the same object. extend disparities into untextured regions. capture disparity discontinuities more precisely. Assign disparities to small disconnected background regions in complex occlusions. 9

10. Outline Introduction Proposed Model Energy Minimization Result Conclusion 10

11. Proposed Model Scene Representation, assume that disparity map is a collection of 3D planes (depth planes). estimate object’s depth by a 3D plane (object plane). compute a parallax value obtained by subtracting p’s disparity at each pixel p within an object o p. 11

12. Parallax Model Enforce parallax values have a compact distribution within object o p. The parallax model provides the probability of the occurrence of a specific parallax in object o p. The proposed model avoid parallaxes that have low probabilities. 12

13. An object o ∈ O contains the following parameters: 1.a color model 2.a parallax model 3.an object plane F : I → F that assigns each pixel to a depth plane.. O : I → O that assigns each pixel to an object. Energy Function 13

14. Energy Function Energy function evaluates the quality of F and O. Minimize the energy to obtain a “good” approximation to the Maximum a Posteriori (MAP) solution of the model.. 14

15. Photo Consistency Term E pc. Measures the pixel dissimilarity of corresponding points and accounts for occlusion handling. Ensures that corresponding pixels are assigned to the same depth plane and object. 15

16. Photo Consistency Term E pc. 16

17. Object-Coherency Term E oc. Encourages neighboring pixels in the image to take the same object label.. [19] 17 [19] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 23:309–314, 2004.

18. Depth Plane-Coherency Term E dc. Depth plane assignments within an object shall be spatially coherent.. 18

19. Object-Color Term E col. Each object contains a color model implemented as a Gaussian Mixture Model (GMM). The GMM gives the probability that a pixel lies inside the object according to its color value. 19

20. Object-Color Term E col. [19] 20 [19] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 23:309–314, 2004.

21. Object-Parallax Term E par. The disparity at pixel p according to o p ’s object plane by. The parallax is then computed as. 21

22. Object-Parallax Term E par Distribution of the parallax within same object is likely to be compact.. 22

23. Object-MDL Term E mdl. The term puts a penalty on the occurrence of an object [4].. 23 [4] M. Bleyer, C. Rother, and P. Kohli. Surface stereo with soft segmentation. In CVPR, 2010.

24. 3D Connectivity E con. An object is considered connected a path connects all pixels with the same object label. The path are either 1.pixels belong to the same object. 2.pixels belong to different objects. 24

25. 3D Connectivity E con. 25

26. Outline Introduction Proposed Model Energy Minimization Proposal Generator Result Conclusion 26

27. Energy Minimization Proposed model is formulated as an energy function that is optimized via fusion moves [16]. In the fusion move, a new solution generated by “selecting” depth planes and objects from S others from S’ 27 [16] V. Lempitsky, C. Rother, and A. Blake. Logcut - efficient graph cut optimization for Markov Random Fields. In ICCV, 2007.

28. Energy Minimization Start with an initial solution S that consists of a disparity map F and an object map O. Obtain a proposal S’ from a proposal generator. S and S’ are fused to produce a new solution S*. S := S* 28

29. Proposal Generator S’ Initial Proposals : initialize the disparity map. color segmentation by mean-shift. derive F, O. estimate parameters. derive a large variety of initial proposals (approximately 30 ). 29

30. Proposal Generator S’ Refit Proposals : compute a new color model, object plane, parallax model. 〈 F’, O’ 〉 is derived by refitting the object parameters of the current solution 〈 F, O 〉. 30

31. Proposal Generator S’ Expansion Proposals : select one depth plane f present in F and one object o present in O. 〈 F’, O’ 〉 is derived by setting all pixels of F’ to f and all pixels of O’ to o. 31

32. Optimal Fusion Use quadratic pseudo-boolean optimization function (QPBO-F) [11] to the fusion move problem. Reduces the problem with multi-valued variables to a sequence of minimization sub-problems with binary variables. 32 [11] V. Kolmogorov and C. Rother. Minimizing non-submodular functions with graph cuts - a review. PAMI, 29(7):1274–1279, 2007.

33. Outline Introduction Proposed Model Energy Minimization Result Conclusion 33

34. Result 34

35. Result 35

36. Result 36

37. Outline Introduction Proposed Model Energy Minimization Result Conclusion 37

38. Conclusion The object level enables our algorithm to utilize color segmentation as a soft constraint and to handle difficult occlusion cases. A 3D connectivity constraint that enforces consistency of object assignments with stereo geometry. Currently, our algorithm is slow, i.e., it takes approximately 20 minutes to obtain results on images. 38