Employing structural representation for symbol detection, symbol spotting and indexation in line drawing document images Muhammad Muzzamil Luqman PhD student (II year) Jean-Yves RamelUniversité François Rabelais de Tours, France Thierry BrouardUniversité François Rabelais de Tours, France Josep LladósUniversitat Autònoma de Barcelona, Spain Thesis supervisors Wednesday, 02 June 2010

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 2 Plan Part 1 Representation and recognition of graphics content in line drawing document images Part 2 Unsupervised indexation and content based (focused) retrieval for line drawing document image repositories

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 3 Plan Part 1 Representation and recognition of graphics content in line drawing document images

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 4 Representation phase Representation of structure of graphics content by an Attributed Relational Graph. Description phase Learning and Classification phase

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 5 Description phase Representation phase Learning and Classification phase Description phase Extraction of signature from ARG.

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 6 Description phase

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 7 Description phase A value laying here fully contributes (i.e. membership weight 1) to the interval “Small” A value laying here contributes in part to the interval “Medium” and in part to the interval “Full”

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 8 Description phase Two iterations over set of ARGs: First iteration 1.Compute ‘connection density counts’ for all ARGs 2.Distribute these ‘connection density counts’ in an optimal number of bins 3.Arrange the bins in a fuzzy fashion to form overlapping intervals for ‘Low’, ‘Medium’ & ‘High’ connection densities. Second iteration Compute signature for graphic symbols (ARGs)

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 9 Learning phase (Structure & Parameters of BN) Representation phase Description phase Learning and Classification phase Encoding of Joint Probability Distribution of signatures by a Bayesian Network. P(Nodes) P(Class|Nodes) P(DenH|DenM)

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 10 Classification phase (Graphics Recognition) Representation phase Description phase Learning and Classification phase Encoding of Joint Probability Distribution of signatures by a Bayesian Network. Bayesian probabilistic inference for recognition. Bayes rule: where Query is recognized as class which gets highest posterior probability!

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 11 Example images

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 12 Noise and deformations 2D linear model symbols from GREC databases Learning on clean symbols and testing against noisy and deformed symbols Results presented in CIFED2010 – With Fuzzy Intervals Results presented in ICDAR2009 – Without Fuzzy intervals

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 13 Noise and deformations 2D linear model symbols from GREC databases Learning on clean symbols and testing against noisy and deformed symbols Comparing results with (Qureshi et al., 2007) and (Luqman et al., 2009)

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 14 Context noise 2D linear model symbols from GREC databases (SESYD dataset) Learning on clean symbols and testing against context-noise Results presented in CIFED2010 – With Fuzzy Intervals

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 15 Some remarks Based on vectorization and hence is sensitive to noise and deformation (which produce irregularities in signature). The proposed signature is more vulnerable to symbols that are composed of circles/arcs. However, lightweight signature and use of an efficient classifier makes it suitable to be used as a pre-processing step to reduce search space or as a quick discrimination method for sufficiently large number of graphic symbols … an application to symbol spotting!

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 16 Generalizing fuzzy signature - Explicit Graph Embedding Vector for explicit embedding of attributed graphs Fuzzy zones for “features for node degrees” (for example) A value laying here contributes in part to the interval “F i2 ” and in part to the interval “F i3 ”

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 17 ICPR2010 contest on Explicit Graph Embedding (GEPR) ICPR2010 contest Graph Embedding for Pattern Recognition (GEPR) Results on sample contest data ALOI (Performance Index: 0.379) COIL (Performance Index: 0.376) ODBK (Performance Index: 0.353) ALOI - Amsterdam Library of Object Images COIL - Columbia Object Image Library ODBK - Object Databank Performance Index measures the quality of clustering (that could be obtained for the embedded vectors). The closer it gets to zero the better the embedding results are!

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 18 Plan Part 2 Unsupervised indexation and content based (focused) retrieval for line drawing document image repositories

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 19 A Symbol Spotting & Focused Retrieval System

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 20 A Symbol Spotting & Focused Retrieval System Unsupervised indexation of line drawing document images  Represent document images by attributed relational graphs  Spot Regions Of Interest (ROIs) in the ARG of document image  Learn parameters for fuzzy structural signature from the set of ROIs  Describe each ROI by a fuzzy structural signature  Cluster signatures of ROIs  Prepare an index (clusterID vs ROIs vs documentImage) and  Learn a BN

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 21 A Symbol Spotting & Focused Retrieval System Content based focused retrieval for line drawing document images  Represent query ROI by attributed relational graph  Spot Regions Of Interest (ROIs)  Describe each query ROI by a fuzzy structural signature  Classify query ROIs using BN and  Retrieve documents using repository index

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 22 A Symbol Spotting & Focused Retrieval System

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 23 Experimentation Dataset SESYD (Systems Evaluation SYnthetic Documents) During learning phase our system detected a total of ROIs in electronic diagrams and 4586 ROIs in floorplans, which approximately corresponds to 108% of the symbols in each of the datasets.

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 24 Experimentation Document Retrieval Results Results presented in ICPR2010 Each point in the graph represents the precision and recall values for a query image.

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion - 25 Conclusion and Future work The Overall framework allows to prepare an index for the document repository in an unsupervised fashion, which is a very important contribution. However the underlying method for ROI localization is based on a set of heuristics and does not return a single symbol in most of the cases and needs to be improved. Future lines of work include the designing of a method to replace the manually selected heuristics by automatic learned heuristics for spotting a ROI.

- 26 References  Delalandre et al., “Building synthetic graphical documents for performance evaluation,” in GREC, vol of LNCS, pp. 288–298, Springer,  Delaplace et al., Two evolutionary methods for learning bayesian network structures, in LNAI  Luqman et al., A Content Spotting System For Line Drawing Graphic Document Images, International Conference on Pattern Recognition, 2010, to appear.  Luqman et al., Vers une approche floue d’encapsulation de graphes: application à la reconnaissance de symboles, Colloque International Francophone sur l'Ecrit et le Document, 2010,  Luqman et al., Graphic Symbol Recognition using Graph Based Signature and Bayesian Network Classifier, Tenth International Conference on Document Analysis and Recognition (ICDAR), IEEE Computer Society, 2009, volume 10,  Luqman et al., Employing fuzzy intervals and loop-based methodology for designing structural signature: an application to symbol recognition, Eighth IAPR International Workshop on Graphics RECognition (GREC), 2009, volume 8,  Qureshi et al., Combination of symbolic and statistical features for symbols recognition, in IEEE ICSCN’2007.  Qureshi et al., “Spotting symbols in line drawing images using graph representations,” in GREC, pp. 91–103, 2007.