1. A Performance Characterization Algorithm for Symbol Localization Mathieu Delalandre 1,2, Jean-Yves Ramel 2, Ernest Valveny 1 and Muhammad Muzzamil Luqman 1,2 1 CVC, Barcelona city, Spain 2 LI Laboratory, Tours city, France LaBRI - Partnerships Meeting Bordeaux, France Thursday 14th of October 2010

2. Recognition Spotting r1 r2 r3 sofa skin tub door document database learning database Query By Example (QBE) rank labels Symbol localization systems (recognition and spotting) Spotting/Recognition System Groundtruth Matching localization results with groundtruth Region Of Interest Characterization measures sofa ski n tub door Labels r1 r2 r3 Ranks QBE truth results Learning Performance characterization To make the correspondence in term of localization To compute characterization measures (recall, precision, recognition rates, etc.) A Performance Characterization Algorithm for Symbol Localization

3. groundtruth results groundtruth – Global discrepancy methods Number of missed segmented pixels Position of missed segmented pixels – Local discrepancy methods Number of region in the image Features values of regions A Performance Characterization Algorithm for Symbol Localization Performance evaluation image segmentationobject localization coverage of resultsall imagepart of precision of localizationhigh importanceweak importance semantic matchingweak importancehigh importance Performance evaluation of image segmentation [Zhang’1996] Performance evaluation : image segmentation vs. object localization truth results Single : an object in groundtruth matches only with one detected object. Split : two objects in groundtruth match with one detected object. Merge : an object in groundtruth matches with two detected objects. Performance evaluation of object localization [Delalandre2009] False alarm : a detected object doesn't match with any object in groundtruth. Miss : an object in groundtruth doesn't match with any detected object.

4. groundtruth results groundtruth Layout analysis [Antonacopoulos1999] Text/graphics separation [Liu1997] truth results Single : an object in groundtruth matches only with one detected object. Split : two objects in groundtruth match with one detected object. Merge : an object in groundtruth matches with two detected objects. Performance evaluation of object localization [Delalandre2009] Symbol spotting [Rusinol2009] char and text boxes groundtruth results isothetic polygons groundtruth results Convex hulls A Performance Characterization Algorithm for Symbol Localization False alarm : a detected object doesn't match with any object in groundtruth. Miss : an object in groundtruth doesn't match with any detected object.

5. A Performance Characterization Algorithm for Symbol Localization in a “part of” segmentation problem, how to make the difference between segmentation errors of background with segmentation errors of objects Ways to solve... 1. “naive” : To use thresholds to “reject” some segmentation results (bad...) 2. ideal : To define directed knowledge based approaches to model localization/segmentation algorithms (hard...) 3. intermediate (proposed) : To use “fuzzy-based” approach, to characterize the characterization results according to confidence rate i.e. this is a positive matching between groundtruth and system’s results with a confidence rate of . Open problem with object localization probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities

6. A Performance Characterization Algorithm for Symbol Localization probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities Localization comparison Probability scores Matching algorithm GroundtruthResults probability error detection rate

7. Result point Intersection line Intersection point Groundtruth, gravity center, contours l gi l gr L g ri c probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities Localization comparison Probability scores Matching algorithm GroundtruthResults probability error detection rate A Performance Characterization Algorithm for Symbol Localization

8. s2s2 s1s1 s3s3 g1g1 r g2g2 g3g3 g2g2 s2s2 s3s3 r g1g1 g3g3 s Groundtruth points Result point r gigi s2s2 s1s1 s3s3 g1g1 r g2g2 g3g3 null probabilities, = equidistant case highest probabilities, = nearest points maximum probability, = equality case probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities Localization comparison Probability scores Matching algorithm GroundtruthResults probability error detection rate A Performance Characterization Algorithm for Symbol Localization

9. cases s i = 0 s j = k s j  +  s i = k s i = s j sj= 0si= ksj= 0si= k si +sj= ksi +sj= k 01 ++ 100 probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities Localization comparison Probability scores Matching algorithm GroundtruthResults probability error detection rate A Performance Characterization Algorithm for Symbol Localization gigi gjgj r How to compute the probability between a groundtruth point g i and the result point r, considering the neighboring groundtruth point g j we define - p i the probability r  g i, regarding g j - s i is the scaling factor between g i and r - s j is the scaling factor between g j and r gigi gjgj r gigi gjgj r gigi gjgj r gigi gjgj r sisi sjsj pipi

10. probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities Localization comparison Probability scores Matching algorithm GroundtruthResults probability error detection rate A Performance Characterization Algorithm for Symbol Localization 01234 x Gaussian function 01 1 Probability score function 01 ++ 100 Thus, our probability function must respect the following properties Several mathematics functions could be used (affine, exponential, trigonometric, etc.) we choose a Gaussian based function as it is good model of random distribution

11. probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities Localization comparison Probability scores Matching algorithm GroundtruthResults probability error detection rate A Performance Characterization Algorithm for Symbol Localization s2s2 s1s1 s3s3 g1g1 r g2g2 g3g3 We extend the computation of probability to a neighboring composed of n groundtruth points like this we define - is the set of groundtruth points - s i is the scaling factor between g i and r - are the scaling factors between and r - is the probability r  g i, regarding

12. Groundtruth points Result points r1r1 r2r2 rqrq g1g1 g2g2 gngn … … d g1 =2 d gn =0 d r1 =1 detection rates 0 1 01score error ε single (T s ) alarm (T f ) multiple (T m ) probability error detection rate p1p1 p3p3 p2p2 Groundtruth, gravity centers, contours Result points Highest probabilities Lowest probabilities Localization comparison Probability scores Matching algorithm GroundtruthResults probability error detection rate A Performance Characterization Algorithm for Symbol Localization

13. Qureshi’2008 Drawing levelSymbol level Settingbackgrounds5models16 Datasetimages100symbols2521 Settingbackgrounds5models17 Datasetimages100symbols1340 floorplans diagrams false alarm (T f ) detection rates score error 1 single (T s ) multiple (T m ) max score error 1 detection rates false alarm (T f ) single (T s ) multiple (T m ) max floorplans diagrams Score0,11 Ts0,57 Tf0,31 Tm0,20 Score0,20 Ts0,62 Tf0,06 Tm0,37 max A Performance Characterization Algorithm for Symbol Localization

14. Characterization Groundtruth 1Results 1 probability error detection rate A Performance Characterization Algorithm for Symbol Localization Characterization Groundtruth 2Results 2 probability error detection rate Each result is context dependent, how to compare them ?

15. Characterization GroundtruthResults probability error detection rate A Performance Characterization Algorithm for Symbol Localization Characterization GroundtruthResults probability error detection rate Transform function single detections T s 0 1 01score error (ε) ε g sisi 0 q 0 global score  i ii 1 ++ number of results (q) n 0 0score error (x) 1 1 i(ε)i(ε) 1(ε)1(ε) 2(ε)2(ε)  1 (1)  2 (1) probability error detection rate We compute the difference between a result and self-matching of his groundtruth (g), to make the new results test-independent. q ii =0 =n=1 ++00

16. A Performance Characterization Algorithm for Symbol Localization Qureshi’2008 Drawing levelSymbol level Settingbackgrounds5models16 Datasetimages100symbols2521 Settingbackgrounds5models17 Datasetimages100symbols1340 floorplans diagrams electrical diagrams floorplans 1,00 electrical diagrams floorplans  i(1) = 0.496  i(1) = 0.529 score error (ε) i(ε)i(ε) i(ε)i(ε)

17. Conclusion and perspectives Conclusion A new fuzzy way to evaluate object localization distribution of matching cases regarding a “confidence rate” Experimentation with a real system electrical and architectural drawings, 200 test images, 3821 symbols Perspectives Extending experiments several systems, to add noise, scalability, real datasets