1. Perceptual Organization: Segmentation and Optical Flow

2. Inspiration from psychology The Gestalt school: Grouping is key to visual perception –“The whole is greater than the sum of its parts” http://en.wikipedia.org/wiki/Gestalt_psychology subjective contours occlusion familiar configuration

3. Gestalt grouping factors

4. Emergence http://en.wikipedia.org/wiki/Gestalt_psychology

5. Motion and perceptual organization Even “impoverished” motion data can evoke a strong percept G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. YouTube video

6. Image segmentation

7. The goals of segmentation Obtain primitives for other tasks Perceptual organization, recognition Graphics, image manipulation

8. Goal 1: Primitives for other tasks Group together similar-looking pixels for efficiency of further processing “Bottom-up” process Unsupervised X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.Learning a classification model for segmentation. “superpixels”

9. Image parsing or semantic segmentation: Segments as primitives for recognition J. Tighe and S. Lazebnik, ECCV 2010, IJCV 2013

10. Goal 2: Recognition Separate image into coherent “objects” “Bottom-up” or “top-down” process? Supervised or unsupervised? Berkeley segmentation database: http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ image human segmentation

11. Goal 3: Image manipulation Interactive segmentation for graphics

12. Approaches to segmentation Segmentation as clustering Segmentation as graph partitioning Segmentation as labeling

13. Segmentation as clustering Source: K. Grauman

14. Image Intensity-based clustersColor-based clusters Segmentation as clustering K-means clustering based on intensity or color is essentially vector quantization of the image attributes Clusters don’t have to be spatially coherent

15. Segmentation as clustering Source: K. Grauman

16. Segmentation as clustering Clustering based on (r,g,b,x,y) values enforces more spatial coherence

17. Segmentation as graph partitioning Node for every pixel Edge between every pair of pixels (or every pair of “sufficiently close” pixels) Each edge is weighted by the affinity or similarity of the two nodes w ij i j Source: S. Seitz

18. Measuring affinity Represent each pixel by a feature vector x and define an appropriate distance function small σ large σ Role of σ

19. Segmentation as graph partitioning Break Graph into Segments Delete links that cross between segments Easiest to break links that have low affinity –similar pixels should be in the same segments –dissimilar pixels should be in different segments ABC Source: S. Seitz w ij i j

20. Graph cut Set of edges whose removal makes a graph disconnected Cost of a cut: sum of weights of cut edges A graph cut gives us a segmentation What is a “good” graph cut and how do we find one? A B Source: S. Seitz

21. Minimum cut We can do segmentation by finding the minimum cut in a graph Efficient algorithms exist for doing this Minimum cut example

22. Minimum cut We can do segmentation by finding the minimum cut in a graph Efficient algorithms exist for doing this Minimum cut example

23. Normalized cut Drawback: minimum cut tends to cut off very small, isolated components Ideal Cut Cuts with lesser weight than the ideal cut * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

24. Normalized cut To encourage larger segments, normalize the cut by the total weight of edges incident to the segment The normalized cut cost is: Intuition: big segments will have a large w(A,V), thus decreasing ncut(A, B) Finding the globally optimal cut is NP-complete, but a relaxed version can be solved using a generalized eigenvalue problem w(A, B) = sum of weights of all edges between A and B J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000Normalized cuts and image segmentation.

25. Normalized cut: Algorithm Let W be the affinity matrix of the graph (n x n for n pixels) Let D be the diagonal matrix with entries D(i, i) = Σ j W(i, j) Solve generalized eigenvalue problem (D − W)y = λDy for the eigenvector with the second smallest eigenvalue The ith entry of y can be viewed as a “soft” indicator of the component membership of the ith pixel Use 0 or median value of the entries of y to split the graph into two components To find more than two components: Recursively bipartition the graph Run k-means clustering on values of several eigenvectors

26. Example result

27. Challenge How to define affinities for segmenting highly textured images?

28. Segmenting textured images Convolve image with a bank of filters Find textons by clustering vectors of filter bank outputs J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation" Texton mapImage Filter bank

29. Segmenting textured images Convolve image with a bank of filters Find textons by clustering vectors of filter bank outputs Represent pixels by texton histograms computed over neighborhoods at some “local scale” Define affinities as similarities between local texton histograms J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

30. Pitfall of texture features Possible solution: check for “intervening contours” when computing affinities J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

31. Results: Berkeley Segmentation Engine http://www.cs.berkeley.edu/~fowlkes/BSE/

32. Berkeley Segmentation Engine http://www.cs.berkeley.edu/~fowlkes/BSE/

33. Pro Generic framework, can be used with many different features and affinity formulations Con High storage requirement and time complexity: involves solving a generalized eigenvalue problem of size n x n, where n is the number of pixels Normalized cuts: Pro and con

34. Efficient graph-based segmentation Runs in time nearly linear in the number of edges Easy to control coarseness of segmentations Results can be unstable P. Felzenszwalb and D. Huttenlocher, Efficient Graph-Based Image Segmentation, IJCV 2004Efficient Graph-Based Image Segmentation

35. Segmentation as labeling Suppose we want to segment an image into foreground and background Binary labeling problem Credit: N. Snavely

36. Segmentation as labeling Suppose we want to segment an image into foreground and background Binary labeling problem User sketches out a few strokes on foreground and background… How do we label the rest of the pixels? Source: N. Snavely

37. Binary segmentation as energy minimization Define a labeling L as an assignment of each pixel with a 0-1 label (background or foreground) Find the labeling L that minimizes data term smoothness term How similar is each labeled pixel to the foreground or background? Encourage spatially coherent segments Source: N. Snavely

38. : “distance” from pixel to background : “distance” from pixel to foreground { computed by creating a color model from user- labeled pixels Source: N. Snavely

40. Neighboring pixels should generally have the same labels Unless the pixels have very different intensities : similarity in intensity of p and q = 10.0 = 0.1 Source: N. Snavely

41. Binary segmentation as energy minimization For this problem, we can efficiently find the global minimum using the max flow / min cut algorithm Source: N. Snavely Y. Boykov and M.-P. Jolly, Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images, ICCV 2001Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images

42. Recall: Stereo as energy minimization I1I1 I2I2 D Energy functions of this form can be minimized using graph cuts Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001Fast Approximate Energy Minimization via Graph Cuts W1(i )W1(i )W 2 (i+D(i )) D(i )D(i ) data term smoothness term

43. GrabCut C. Rother, V. Kolmogorov, and A. Blake, “GrabCut” — Interactive Foreground Extraction using Iterated Graph Cuts, SIGGRAPH 2004“GrabCut” — Interactive Foreground Extraction using Iterated Graph Cuts