1. 1 How do ideas from perceptual organization relate to natural scenes?

2. 2 Brunswik & Kamiya 1953 Thesis: Gestalt rules reflect the structure of the natural world Attempted to validate the grouping rule of proximity of similars Brunswik was ahead of his time… we now have the tools. Egon Brunswik (1903-1955)

3. 3 Can we define these cues for real images? Are these cues “ecologically valid”? How informative are different cues? Grouping Figure/Ground Ecological Statistics of Perceptual Organization

4. 4 Task: detect generic pattern or group Signal: class of patterns, known null hypothesis Cues: optimal test is usually obvious Result: mathematically precise characterization of when detection is possible Task: capture “useful” information about the scene Signal: natural image statistics, clutter Cues: something computable from real pixels Result: empirical statistics about relative power of different cues

5. 5 Berkeley Segmentation DataSet [BSDS]

6. 6 Cues: a)distance [proximity] b)region cues [similarity] c)boundary cues [connectedness, closure, convexity] What image measurements allow us to gauge the probability that pixels i and j belong to the same group?

7. 7 Learning Pairwise Affinities S ij – indicator variable as to whether pixels i and j were marked as belonging to the same group by human subjects. W ij – our estimate of the likelihood that pixel i and j belong to the same group conditioned on the image measurements. Use the ground truth given by human segmentations to calibrate cues. Learn “statistically optimal” cue combination in a supervised learning framework Ecological Statistics: Measure the relative power of different cues for natural scenes

8. 8 Color a* b* Brightness L* Texture Original Image W ij Distance E D 22 Boundary Processing Textons A B C A B C 22 Region Processing

9. 9 Evaluation Measures 1.Precision-Recall of same-segment pairs –Precision is P(S ij =1 | W ij > t) –Recall is P(W ij > t | S ij = 1) 2.Mutual Information between W and S Groundtruth S ij Estimate W ij ∫ p(s,w) log [p(s)p(w) / p(s,w)]

10. 10 Individual Features Patches Gradients

11. 11 Affinity Model vs. Human Segmentation

12. 12 Findings Both Edges and Patches provide useful “independent” information. Texture gradients can be quite powerful Color patches better than gradients Brightness gradients better than patches. Proximity is a result, not a cause of grouping

13. 13 Figure-Ground Labeling - start with 200 segmented images of natural scenes - boundaries labeled by at least 2 different human subjects - subjects agree on 88% of contours labeled

14. 14 Local Cues for Figure/Ground Assume we have a perfect segmentation Can we predict which region a contour belongs to based on it’s local shape? –Size/Surroundedness –Convexity –Lower Region

15. 15 Size(p) = log(Area F / Area G ) Size and Surroundedness [Rubin 1921] G F p

16. 16 Convexity(p) = log(Conv F / Conv G ) Conv G = percentage of straight lines that lie completely within region G p G F Convexity [Metzger 1953, Kanizsa and Gerbino 1976]

17. 17 LowerRegion(p) = θ G Lower Region [Vecera, Vogel & Woodman 2002] θ p center of mass

18. 18 Figural regions tend to be convex

19. 19 Figural regions tend to lie below ground regions

20. 20 Size Lower Region Convexity

21. 21 Power of cue depends on support of the analysis window.

22. 22 Power of cue depends on support of the analysis window.

23. 23 “Upper Bounding” Local Performance Present human subjects with local shapes, seen through an aperture.

24. 24 Human Performance on Local Figure-Ground

25. 25 Extension to Real Images Build up library of prototypical contour configurations by clustering local shape descriptors –Geometric Blur [Berg & Malik 01] Train a classifier which uses similarities to these prototype shapes to predict figure/ground label

26. 26 Shapemes Classifier using 64 shapeme features: 61%

27. 27 Globalization of Figure/Ground Measurements Averaging local shapeme cue over human-marked boundaries: 71% Prior over junction types and label continuity: 79%