Performance Evaluation of Grouping Algorithms Vida Movahedi Elder Lab - Centre for Vision Research York University Spring 2009

Centre for Vision Research, York University2 Overview  Grouping and evaluation methods  Region-based measures  Boundary-based measures  Mixed measures  Alignment measure

Centre for Vision Research, York University3 Overview Grouping and evaluation methods  Region-based measures  Boundary-based measures  Mixed measures  Alignment measure

Centre for Vision Research, York University4 Grouping Edge segments: Example

Centre for Vision Research, York University5 Perceptual Organization/ Grouping  A process of assembling features into groups which are perceptually significant based on various cues (Lowe, 1985)  The problem of aggregating primitive image features that project from a common structure in the visual scene (Elder, 2002)

Centre for Vision Research, York University6 Evaluation Measure  How good is each grouping?  Which algorithm has a better performance?  What is the best grouping that can be achieved?  Note differences with regional segmentation evaluation

Centre for Vision Research, York University7 Evaluation Methods  Three main categories (Zhang, 1996) Analytical methods Consider the algorithms themselves, e.g. based on the a priori knowledge they use (not based on output of the algorithms) Empirical goodness methods Based on the outputs of the algorithms, e.g. based on the intra-region uniformity of the segments, or the inter-region contrast between the segments. Empirical discrepancy methods A reference segmentation or ground truth is assumed, to compare the outputs with

Centre for Vision Research, York University8 Goal: Measure Discrepancy

Centre for Vision Research, York University9 SOD: Salient Object Dataset  Based on Berkeley Segmentation Dataset (BSD)  300 images  7 subjects

Centre for Vision Research, York University10 Overview Grouping and evaluation methods Region-based measures  Boundary-based measures  Mixed measures  Alignment measure

Centre for Vision Research, York University11 Region-based Discrepancy  (Young, 2005),(Ge, 2006), (Goldmann, 2008)  A and B two boundaries  R B the region corresponding to a boundary B and |R B | the area of this region  1: maximum discrepancy,  0: maximum consistency

Centre for Vision Research, York University12 Interpretation

Centre for Vision Research, York University13 Evaluation by this measure  Not sensitive to spikes, wiggles, shape >= (more error)

Centre for Vision Research, York University14 Examples of near-optimal cases

Centre for Vision Research, York University15 Overview Grouping and evaluation methods Region-based measures Boundary-based measures  Mixed measures  Alignment measure

Centre for Vision Research, York University16 Distance of one point a from B Distance Signature of all a in A One directional Hausdorff Two directional Hausdorff Boundary-based Distance

Centre for Vision Research, York University17 Evaluation by this measure  Not sensitive to wiggles, shape  Not sensitive to the distance distribution, but only to the maximum value

Centre for Vision Research, York University18 Geodesic Distance: the min. distance between two points a and b without cross Euclidean vs. Geodesic Distance Euclidean Distance: the min. distance between two points a and b

Centre for Vision Research, York University19 Evaluation by this measure Almost the same by D e Almost the same by D e & D g

Centre for Vision Research, York University20 Overview Grouping and evaluation methods Region-based measures Boundary-based measures Mixed measures  Alignment measure

Centre for Vision Research, York University21 A mixture of boundary-based and region-based  Penalizing the over-detected and under-detected regions by their Euclidean or Geodesic distances p j, j=1..N fp are pixels in the false negative region (R B -R A ) q k, k=1..N fn are pixels in the false positive region (R A -R B )

Centre for Vision Research, York University22 Evaluation by this measure  Not penalizing effectively, e.g. narrow false positives below

Centre for Vision Research, York University23 Correspondence Problem  The false negative and false positive regions can be very small, yet the boundaries be very different  Segments on one boundary should correspond to segments on the other

Centre for Vision Research, York University24 Alignment  The order of matching points on the two boundaries should be monotonically non-decreasing.

Centre for Vision Research, York University25 Correspondence (Cont.)  Note that if correspondence is maintained, D e will work almost like D g !

Centre for Vision Research, York University26 Overview Grouping and evaluation methods Region-based measures Boundary-based measures Mixed measures Alignment measure

Centre for Vision Research, York University27 Alignment Distance  Main idea: We need to find the ‘alignment’ that leads to minimum total distance.  Method: Use N samples on each boundary (equally spaced) Find the NxN matrix of Euclidean distances. The diagonals show correspondences with some rotations The one with min sum of distances is the best correspondence and its sum is our measure of discrepancy.

Centre for Vision Research, York University28 Alignment Measure (cont.) Note: Order of both samples increases clockwise

Centre for Vision Research, York University29 Evaluation by this simple measure  Samples falling out of phase  Solution: finer sampling on one boundary >= (more error)

Centre for Vision Research, York University30 Bimorphism  (Tagare, 2002)  A method to let correspondence of 1 to many and many to 1  symmetric

Centre for Vision Research, York University31 A symmetric Alignment Distance  Edit cost of changing one string to another  Edit operation, cost of operation  A sequence of operations taking A to B  Symmetric:

Centre for Vision Research, York University32 Example

Centre for Vision Research, York University33 Cyclic shifts  Cyclic shifts  Alignment Distance  Dynamic programming  Complexity:  (Maes, 1990) Complexity:

Centre for Vision Research, York University34 Examples

Centre for Vision Research, York University35 Examples

Centre for Vision Research, York University36 Evaluation by this measure  Note: If using Euclidean distance, there is no sensitivity to region

Centre for Vision Research, York University37 References (Elder, 2002) J. H. Elder and R. M. Goldberg (2002), "Ecological statistics of Gestalt laws for the perceptual organization of contours." J Vis, vol. 2, pp (Zhang, 1996) Y. J. Zhang. (1996), “A survey on evaluation methods for image segmentation”, Pattern recognition 29(8), pp (BSD) D. Martin (2001), "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics," Proceedings of the 8th IEEE International Conference on Computer Vision, vol. 2, pp (Ge, 2006) F. Ge, S. Wang and T. Liu (2006), "Image-Segmentation Evaluation From the Perspective of Salient Object Extraction," Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, pp (Goldmann, 2008) L. Goldmann. (2008), Towards fully automatic image segmentation evaluation. Lecture notes in computer science 5259 LNCS, pp (Young, 2005) D. P. Young (2005), "PETS Metrics: On-line performance evaluation service," Proceedings - 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, VS-PETS, vol. 2005, pp. 317, (Huttenlocher, 1993) D. P. Huttenlocher (1993), “Comparing images using the Hausdorff distance”, IEEE transactions on pattern analysis and machine intelligence 15(9), pp (Tagare, 2002) H. D. Tagare. (2002), “Non-rigid shape comparison of plane curves in images”, Journal of mathematical imaging and vision 16(1), pp. 57. (Maes, 1990) M. Maes (1990), “On a cyclic string-to-string correction problem”, Information processing letters 35(2), pp. 73.