1. Object Segmentation Presented by Sherin Aly 1

2. What is a ‘Good Segmentation’?

3. http://www.eecs.berkeley.edu/Research/Projects/CS/vision/groupin g/resources.html

4. Learning a classification model for segmentation Xiaofeng Ren and Jitendra Malik 4

5. methodology Two-class classification model Over segmentation as preprocessing They use classical Gestalt cues –Contour, texture, brightness and continuation A linear classifier is used for training 5

6. Good Vs Bad segmentation 6 a) Image from Corel Imagebase b) superimposed with a human marked segmentation c) Same image with Bad segmentation

7. How do we distinguish good segmentations from bad segmentations? 7

8. How? Use “Classical Gestalt cues” – proximity, similarity and good continuation Instead of Ad-hoc decision about features combination 8

9. Gestalt Principles of Grouping 9 http://allpsych.com/psychology101/perception.html In order to interpret what we receive through our senses,we attempt to organize this information into certain groups.

10. Methodology Preprocessing Feature extraction Feature evaluation Training Optimization Find good segmentaion 10

11. Preprocessing 11 Superpixel map K=200 Reconstruction of human segmentation from Superpixels a contour-based measure is used to quantify this approximation Local Coherent Preserve structure Contour texture

12. 12 The percentage of human marked boundaries covered by the superpixel maps Tolerance 1,2,and 3

13. Feature Extraction 1. inter-region texture similarity 2. intra-region texture similarity 3. inter-region brightness similarity 4. intra-region brightness similarity 5. inter-region contour energy 6. intra-region contour energy 7. curvilinear continuity 13

14. Feature Extraction 1. inter-region texture similarity 2. intra-region texture similarity 3. inter-region brightness similarity 4. intra-region brightness similarity 5. inter-region contour energy 6. intra-region contour energy 7. curvilinear continuity 14

15. Feature Extraction 1. inter-region texture similarity 2. intra-region texture similarity 3. inter-region brightness similarity 4. intra-region brightness similarity 5. inter-region contour energy 6. intra-region contour energy 7. curvilinear continuity 15

16. Power of Gestalt cues 16 =

17. Training the classifier simple logistic regression classifier, 17 Empirical distribution of pairs of features

18. 18 Precision is the fraction of detections which are true positives. Recall is the fraction of true positives which are detected

19. Conclusion There simple linear classifier had promising results on a variety of natural images. boundary contour is the most informative grouping cue, and it is in essence discriminative. 19

20. Pros & Cons Cons –The larger spatial support that superpixels provide, allowing more global features to be computed than on pixels alone. –The use of superpixels improves the computational efficiency –SuperPixels technique is very applicable Pros –Might fall in Local Minima 20

21. Combining Top-down and Bottom-up Segmentation Eran Borenstein Eitan Sharon Shimon Ullman 21

22. Motivation Bottom-Up segmentation –Rely on continuity principle –Capture image properties “texture, grey level uniformity and contour continuity” –Segmentation based on similarities between image regions How can we capture prior knowledge of a specific object (class)? –Answer: Top-Down Segmentation –use prior knowledge about an object Credit: Joseph Djugash

23. Bottom-Up Segmentation Slides from Eitan Sharon, “ Segmentation and Boundary Detection Using Multiscale Intensity Measurements ”. Credit: Joseph Djugash

24. Normalized-Cut Measure Slides from Eitan Sharon, “ Segmentation and Boundary Detection Using Multiscale Intensity Measurements ”. Credit: Joseph Djugash

25. Top-Down approach Input Fragments MatchingCover Credit: Joseph Djugash

26. Another step towards the middle Bottom-Up Top-Down Credit: Joseph Djugash

27. Some Definitions & Constraints Measure of saliency h(Γ i ), h i є [0,1) A configuration vector s contains labels s i (1/- 1) of all the segments (S i ) in the tree The label s i can be different from its parent’s label s i – Cost function for a given s Top-down termBottom-up term Defines the weighted edge between S i & S i –

28. Classification Costs The terminal segments of the tree determine the final classification The top-down term is defined as: The saliency of a segment should restrict its label (based on its parent’s label) The bottom-up term is defined as:

29. Minimizing the Costs – Information Exchange in a Tree Bottom-up message: Top-down message: Min-cost Label: Cost of s i = –1 and s = x Message from s i = –1 Cost of s i = +1 and s = x Message from s i = +1 Computed at each node – minimal of the values is the selected label of node s in s Minimal Cost if the region was classified as background Minimal Cost if the region was classified as figure

30. Confidence Map Evaluating the confidence of a region: Causes of Uncertainty of Classification –Bottom-up uncertainty – regions where there is no salient bottom-up segment matching the top-down classification –Top-down uncertainty – regions where the top- down classification is ambiguous (highly variable shape regions) The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation

31. Results Calculate average distance between a given segmentation contour and a benchmark contour. Removing from the average all contour points having a confidence measure less than 0.1. The resulting confidence map efficiently separated regions of high and low consistency. The combined scheme improved the top-down contour by over 67% on average. This improvement was even larger in object parts with highly variable shape. 31

32. Results (cont.) top-down process may produce a figure-ground approximation that does not follow the image discontinuities. Salient bottom-up segments can correct these errors and delineate precise region boundaries Buttom up The initial classification map T(x, y)

33. Results III (cont.)

34. the top-down completely misses a part of the object. The confidence map may be helpful in identifying such cases,

35. Results III (cont.) bottom-up segmentation may be insufficient in detecting the figure-ground contour, and the top-down process completes the missing information

36. Results III (cont.)

37. Salient bottom-up segments can correct these errors and delineate precise region boundaries

38. Conclusion Buttom-up and top-down merits Provide reliable confidence map It take into account all discontinuities at all scales But: If the object is assigned a given category, the specific features cannot be adopted for other categories 38

39. Constrained Parametric Min-Cuts for Automatic Object Segmentation Joao Carreira Cristian Sminchisescu 39

40. Traditional Segmentation: Finding Homogeneous Regions 40 gPb-owt-ucm: P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2010.

41. Conventional Bottom-up Segmentation Proposed approach 1.Split multiple times 2.Retain object-like segmentations Bottom-up Object Segmentation Credit: J. Carreira High redundancy

42. Bottom-up Object Segmentation 42 Credit: J. Carreira A single multi-region segmentation or a hierarchy

43. Proposed Bottom-up Object Segmentation 43 Credit: J. Carreira single-shot multi- region segmentation robust set of overlapping figure-ground segmentations Segments with object-like regularities superpixels

44. 44 Constrained Parametric Min-Cuts for Automatic Object Segmentation Credit: J. Carreira parametric max-flow solver Figure ground segmentation by growing regions around seeds Ranking

45. 45 Constrained Parametric Min-Cuts for Automatic Object Segmentation Credit: J. Carreira

46. Initialization Foreground –Regular 5x5 grid geometry –Centroids of large N-Cuts regions –Centroids of superpixels closest to grid positions Background –Full image boundary –Horizontal boundaries –Vertical boundaries –All boundaries excluding the bottom one Performance broadly invariant to different initializations

47. Generating a segment pool: c onstrained min-cut min cut hard constraint background object hard constraint 47 Credit: J. Carreira

48. Generating a Segment Pool: C onstrained Parametric Min-Cuts 48 Credit: J. Carreira

49. 49 Generating a Segment Pool: C onstrained Parametric Min-Cuts Credit: J. Carreira

50. 50 Generating a Segment Pool: C onstrained Parametric Min-Cuts Credit: J. Carreira

51. Can solve for all values of object bias in the same time complexity of solving a single min-cut using a parametric max-flow solver 51 Generating a Segment Pool: C onstrained Parametric Min-Cuts Credit: J. Carreira

52. Fast Rejection Large set of initial segmentations (~5500) High Energy Low Energy ~2000 segments with the lowest energy Cluster segments based on spatial overlap (at least 0.95) Lowest energy member of each cluster (~154 in PASCAL VOC) Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

53. 53 Constrained Parametric Min-Cuts for Automatic Object Segmentation Credit: J. Carreira ranks all the sampled object segmentations discard all but a small subset of confident ones.

54. Ranking object hypotheses mid-level, category independent f eatures  Boundary – normalized boundary energy  Region – location, perimeter, area, Euler number, orientation, contrast with background  Gestalt – convexity, smoothness Good Low boundary energy Non smooth. High Euler number High boundary energy Smooth. Euler number = 0 Bad 54 Credit: J. Carreira

55. Traditional Classification-based Learning 55 Credit: J. Carreira

56. Traditional Classification-based Learning 56 Credit: J. Carreira

57. Proposed Ranking-based Learning 57 Credit: J. Carreira

58. Segment Ranking Model data using a host of features –Graph partition properties –Region properties –Gestalt properties Apply Features Normalization Train regressor with the largest overlap ground-truth segment using Random Forests Diversify similar rankings using Maximal Marginal Relevance (MMR)

59. Graph Partition Properties Cut – Sum of affinities along segment boundary Ratio Cut – Sum along boundary divided by the number Normalized Cut – Sum of cut and affinity in foreground and background Unbalanced N-cut – N-cut divided by foreground affinity Thresholded boundary fraction of a cut

60. Region Properties Area Perimeter Relative Centroid Bounding Box properties Fitting Ellipse properties Eccentricity Orientation Convex Area Euler Number Diameter of Circle with the same area of the segment Percentage of bounding box covered Absolute distance to the center of the image

61. Gestalt Properties Inter-region texton similarity Intra-region texton similarity Inter-region brightness similarity Intra-region brightness similarity Inter-region contour energy Intra-region contour energy Curvilinear continuity Convexity – Ratio of foreground area to convex hull area

62. Feature Importance for the Random Forest regressor

63. Feature Importance

64. How to Model Segment Quality ? Best overlap with a ground truth object computed by intersection-over-union. 64 Credit: J. Carreira

65. What has been modeled? Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

66. Diversifying the Ranking Diversified Original Best two hypotheses Middle two hypotheses Worst two hypotheses Segment Ranking using Maximum Marginal Relevance 66

67. Databases –Weizmann database F-measure criterion –MSR-Cambridge database & Pascal VOC2009 Segmentation covering

68. Performance Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

69. Test of the algorithm Berkeley segmentation dataset –Complete pool of images collected –Ranked using the ranking methodology –Top ranks evaluated to test the ranking procedure How well does the algorithm perform? Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

70. Berkeley Database Rank 269! Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

71. Berkeley Database Rank 142! Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

72. Berkeley Database Rank 98! Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

73. Berkeley Database Compute the Segment Covering score for the top 40 segments of each image in the database DatabaseSegment Covering Score (Top 40) BSDS0.52 MSR Cambridge0.77 Pascal VOC0.63 DatabaseSegment Covering Score (All segments) BSDS0.61 MSR Cambridge0.85 Pascal VOC0.78

74. Conclusion Does Constrained Parametric Min-Cuts work well? –Yes Does Fast Rejection work well? –Yes Does Segment Ranking work well? –I don’t think so Credit: SasiKanth Bendapudi Yogeshwar Nagaraj

75. Image Ground-truth objects Best in segment pool Best in top- ranked 200 75 CPMC Segmentation Examples

76. CPMC Results gPb-owt-ucm: P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2010. Equals the state-of-the-art on the VOC 2009 dataset using just 7 segments. 76 VOC2009CoveringNumber of Segments CPMC0.78154 gPb-owt-ucm0.611286 MSRCCoveringNumber of Segments CPMC0.8557 gPb-owt-ucm0.78670

77. Ranking Diversification Diversified Original First two hypothe ses Middle two hypothes es Last two hypothes es 77 Credit:Carreira

78. 78 Credit: J. Carreira

79. Ranking 79 Credit: J. Carreira

80. Running Demos Methodologies employed –Kmeans using: Texture RGB Texture + RGB RGB + HSV Texture + Lab + HSV 80

81. Running Demos Data set used –Microsoft Research Cambridge Object Recognition Image Database, version 1.0. –Used: 7 classes with 23 per class Animal-grass Trees-sky-grass Buildings-sky-grass Airplanes-sky-grass Animal-grass Faces-BG Car-wall-ground 81

82. Experiment Results FeaturesTextureTexture + RGB RGBRGB +HSVTexture+Lab+ HSV Animal-grass72.7%74.1% 72.3%72.6%74.1% Trees-sky- grass 37.1% 40.7%38.2%37.1% Buildings-sky- grass 44.6%42.8% 51.9% 45.4% 44.7% Airplanes-sky- grass 58.8% 54.6% 59.7% 58.7% Animal-grass64.8% 69.3% 71% 64.9% Faces-BG 100% Car-wall-ground 67.2% 68.4% 64.9% 67.2% Mean63.6%63.5%65.3%64.6%63.8% 82

83. Experiment Results FeaturesTextureTexture + RGB RGBRGB +HSVTexture+Lab+ HSV One iteration Elapsed time is 7.42 secs 12.26 secs. 1.62 secs 1.5 secs7.84 sec Overall Elabsed time for experiment 19.9 mins 32.9 mins 4.4 mins4 min21 mins 83 Microsoft Research Cambridge Object Recognition Image Database, version 1.0.

84. 84

85. 85

86. 86

87. 87

88. Acknowledgment Dr. Devi Parikh Dr. Joao Carreira 88