Mathieu Delalandre1, Ernest Valveny1, Tony Pridmore2 Graphics Programming of Constraint for Object Positioning: Application to Performance Evaluation of Symbol Recognition & Spotting Systems Mathieu Delalandre1, Ernest Valveny1, Tony Pridmore2 1 CVC, Barcelona, Spain 2 SCSIT, Nottingham, UK CVC Seminar CVC, Barcelona, Spain Friday 27th of June 2008

Introduction Electrical diagram Mechanical drawing Utility map Recognition Spotting r1 r2 r3 sofa skin tub door document database learning database Query By Example (QBE) rank labels Symbol spotting: “a way to efficiently localize possible symbols and limit the computational complexity, without using full recognition methods” [Tombre2003] [Dosch2004] [Tabbone2004] [Zuwala2006] [Locteau2007] [Qureshi2007] [Rusinol2007] Symbol recognition: ““a particular application of the general problem of pattern recognition, in which an unknown input pattern (i.e. input image) is classified as belonging to one of the relevant classes (i.e. predefined symbols) in the application domain” [Chhabra1998][Cordella1999] [Llados2002] [Tombre2005] scanned CAD file Web image symbol background text

Introduction Characterisation Groundtruth Groundtruthing Results Performance evaluation System Performance evaluation: Information Retrieval [Salton1992], Computer Vision [Thacker2005], CBIR [Muller2001], DIA [Haralick2000]

Introduction Groundtruthing methods for symbol recognition/spotting: Real approach Document Groundtruth Groundtruthing Quickness Realism Reliability Number Connected Noise-effect [Dosch’06 - - ++ - many yes no Yan’04 Aksoy’00 Zhai’03 one Valveny’07 - - weak ++ good real approach synthetic approach GT ground-truthing validation groundtruth drawings and alerts groundtruthed drawings and alerts evaluation test images recognition results Dosh and al 2006 1 4 3 2 5 connected parallel and overlapped Yan and al 2004

Introduction Groundtruthing methods for symbol recognition/spotting: Synthetic approach Document Groundtruth Groundtruthing Setting Quickness Realism Reliability Number Connected Noise-effect [Dosch’06 - - ++ - many yes no Yan’04 Aksoy’00 Zhai’03 one Valveny’07 - - weak ++ good real approach synthetic approach Aksoy 2000 binary noise vectorial noise Valveny and al 2007 Zhai and al 2003

Symbol recognition people Introduction Problematic, how to build whole synthetic documents ? symbol background Graphical documents are composed of two layers To use a same background layer with different symbol layers To associate symbols to positioning constraints and to select them at random Key appraoch c2 c1 First constraint, how to deal with the knowledge level ? - + 2-connected 1-connected Production rules 0-n 1 O-n ‘Creasy’ but well formed drawing Second constraint, how to make usable the system ? Symbol recognition people Database chemical symbol network symbols

Plan Positioning constraint Building system Graphics User Interface 4. Experiments 5. Conclusions and perspectives Positioning constraint Building system Graphics User Interface Experiments Conclusions and perspectives

Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives How to define the constraint ? Model dependent constraint a model constraints to include to connect to border Zone Our approach, model independent control point anchor point selection (3) positioning on background (1) computation of control point (2) sloping +1 -1 ρ = 1 ;  = 0.125 ρ  Unit polar coordinate to define a control point inside a bounding box (2) (3) (4) L d (1) How to get the length l from ρ ? dx dy (2) (3) (4) d (1) dx ϴ 1 dy How to get the direction d from θ ?

Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives (a) (b) 1 0,5 l = r  L L b e r p l point 01 The polygon includes the point if | 01 + 12 + …… 60 | = 2π 3 4 5 6 2 12 + - d1 d2 2π -  d1 and d2 are the two lines’ directions How to select the anchor point ? x0 y0 y0+dy x0+dx x0+rxdx y0+rydy 0  rx,y  1 3. Taken at random inside a zone 1. Just a fixed value (x,y) 1 0,5 r 2. Taken at random belong a line x,y x’,y’ dx dy  l 4. Other shapes (arc, curve, ellipsis etc.)

Building system 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives model selection full constraint overflow constraint selection symbol factory & positioning overlap continue constraint stack empty building failure symbol and constraint object and constraint no yes building end new loop (1) symbol and constraint selection (2) symbol factory and positioning (3) constraint checking (5) Stopping criterion (4) building checking

Building system constraint based selection symbol based selection 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives model selection full constraint overflow constraint selection symbol factory & positioning overlap continue constraint stack empty building failure symbol and constraint object and constraint no yes building end new loop S1 S2 S3 S4 C1 c2 c1 C2 C3 C4 (1) symbol and constraint selection constraint based selection ci s1 s2 s3 ci-1 ci-2 … wc1 = 0.33 wc2 = 0.33 wc3 = 1.00 ws1 = 0.33 ws2 = 0.33 ws3 = 0.66 ws4 = 0.33 ws5 = 0.33 ws6 = 1.00 ps1 = 0.11 ps2 = 0.11 ps3 = 0.22 ps4 = 0.11 ps5 = 0.11 ps6 = 0.33 s1 s2 s3 s4 s5 s6 c1 c2 c3 (c)onstraint and (s)ymbol symbol based selection

Building system (3) sloping rotation scaling, morphing (2) computation 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives model selection full constraint overflow constraint selection symbol factory & positioning overlap continue constraint stack empty building failure symbol and constraint object and constraint no yes building end new loop begin end width center radius x,y x’,y’ dx dy  l arc is directed to  because  is lower than γ 2Π γ  trigonometric d1 d2 2π - d1 and d2 are the two lines’ directions (2) symbol factory and positioning control point anchor point selection (4) positioning on background (2) computation of control point rotation scaling, morphing (3) sloping

Building system Full constraint 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives model selection full constraint overflow constraint selection symbol factory & positioning overlap continue constraint stack empty building failure symbol and constraint object and constraint no yes building end new loop L covers L if e and b are overlapped b e L left bottom right up symbol overflows L if right or up or left or bottom is not covered by L L overlap p if dx1  dx2 < 0 or dy1  dy2 < 0 dx1 dx2 dy1 dy2 p p1 p2 p3 p4 symbol overflows P if P doesn’t include p1 or p2 or p3 or p4 P (a) (b) 1 0,5 l = r  L L b e r p l point 01 The polygon includes the point if | 01 + 12 + …… 60 | = 2π 3 4 5 6 2 12 + - (3) constraint checking Full constraint Each constraint have a number of symbol to push

Building system 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives model selection full constraint overflow constraint selection symbol factory & positioning overlap continue constraint stack empty building failure symbol and constraint object and constraint no yes building end new loop Layer 2 with overlap Layer 1 without overlap background L overlap p if dx1  dx2 < 0 or dy1  dy2 < 0 dx1 dx2 dy1 dy2 p L L overlaps L if e or b are overlapped b e Boxes are overlapped if L1 overlaps L2 or L2 overlaps L1 and L3 overlaps L4 or L4 overlaps L3 L1 L2 L3 L4 (4) building checking

Contributions Building Manager 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives model selection full constraint overflow constraint selection symbol factory & positioning overlap continue constraint stack empty building failure symbol and constraint object and constraint no yes building end new loop Continue When building failures (overflow, full, overlap) become upper to number of wished symbol we stop the process (5) Stopping criterion

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Symbol Models Building Engine Run View Edit Constraint Setting Model Selector Constraint Selector Model Viewer Building Bar Constraint Locator Building Viewer 1. Background broswing 2. Model loading 3. Model viewing 4. Constraint edit 5. Constraint linking 6. Building viewing 7. Constraint import/eport 8. Resolution setting …

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives attach focus ‘click and well’ shift ‘drag’ Background attachment and browsing Model loading, selection and viewing loading select ‘scroll & click’ automatic zooming

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives ‘click’ ‘click, move, click’ or’ ‘pressed, drag, release’ ‘click(s), double click’ Locate the constraints select ‘click’ link model add ‘right double click’ delete ‘left double click’ Link the constraints

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Control Slope

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives

Experiments 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives dataset images symbols models bags #16 1600 15046 25-150 floorplans #10 1000 26830 16 #26 2600 41876

Experiments without rotation, without scaling with rotation, 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives dataset images symbols models bags #16 1600 15046 25-150 floorplans #10 1000 26830 16 #26 2600 41876 without rotation, without scaling with rotation, without scaling without rotation, with scaling with rotation, with scaling

Experiments 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives dataset images symbols models bags #16 1600 15046 25-150 floorplans #10 1000 26830 16 #26 2600 41876

Experiments 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives dataset images symbols models bags #16 1600 15046 25-150 floorplans #10 1000 26830 16 #26 2600 41876

Conclusions and Perspectives 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Conclusions First valid system to build whole graphic synthetic documents Built documents kept meaning and seem realistic No constraint on database sizes (thousands …..) Contents of documents can be controlled + Can be use on several backgrounds with “reasonable” work costs using the GUI (1-2 hour per background)

Conclusions and Perspectives 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives In progress, perspectives Evaluation objectives Campaign organization Background types Model & domain scalability Sizes of image and databases Degradation models System training (learning data, QBE) Groundtruth data & file Mapping algorithm Recognition & ranking System taxonomy System profiling Forum document LITIS Rouen CVC Barcelona SCSIT Nottingham LI Tours LORIANancy Alicia Fornes, Dimosthenis Karatzas, Hervé Locteau, Jean-Pierre Salmon, Jean-Yves Ramel, Marçal Rusinol, Philippe Dosch, Rashid Qureshi and Tony Pridmore Workgroup Spotting/Recognition System Groundtruth Mapping Region Of Interest Characterization sofa skin tub door Labels r1 r2 r3 Ranks QBE truth results Learning

Thanks Bibliography M. Delalandre, E. Valveny and J. Lladós Performance Evaluation of Symbol Recognition and Spotting Systems: An Overview. Accepted paper to the Eighth IAPR Workshop on Document Analysis Systems (DAS), 2008. M. Delalandre, T. Pridmore, E. Valveny, H. Locteau and E. Trupin. Building Synthetic Graphical Documents for Performance Evaluation. Revised Selected Papers of Workshop on Graphics Recognition (GREC), Lecture Notes in Computer Science (LNCS), vol 5046, pp 288-298, 2008.