From Web Documents to Old Books Works in Progress in Graphics Recognition Mathieu Delalandre Meeting of Document Analysis Group Computer Vision Center Barcelona, Spain Thursday 23th November 2006

Plan Short CV Vector Graphics Indexing and Retrieval Dropcap Image Retrieval

LITIS Rouen CVC Barcelona SCSIT Nottingham L3i La Rochelle Short CV Personal Information Mathieu Delalandre, 32 years old Academic Degrees Lic.Sc in Electronic Rouen University, France M.Sc in Industrial Computing Rouen University, France Research Periods LengthPosition Laboratory Subject 6 monthsMaster LITIS symbol recognition 3 ½ yearsPhD LITIS drawing understanding 5 monthsPost-doc SCSIT vector graphics indexing 13monthsPost-doc L3i dropcap image retrieval 2 monthsContract LITIS performance evaluation 3 yearsPost-doc CVC …

Plan Short CV Vector Graphics Indexing and Retrieval Dropcap Image Retrieval

<rect x="400" y="100" width="400“ height="200" fill="yellow" stroke="navy" stroke-width="10" /> What are vector graphics ? Bitmap vs vector graphics More accurate and lighter Known vector graphics formats AI (Adobe Illustrator) SVG (Scalable Vector Graphic) WMF (Windows Metafile) EPS (Encapsulated PostScript) DXF (AutoCAD) ClipArt WMF penEPS Plane Clipart cheese Vector graphics are growing on Web 2001SVG SVG widely used structured documents [Mong’03], geographic maps [Chen’04], technical drawings [Kang’04] 2005Powerful editors (Inskape, Webdraw, …) 2006Internet Explorer and Mozilla Firefox support SVG Application of vector graphics 1982Computer Aided Design (DXF ‘1982’) 1985Office software (PS ‘1985’, CGM ‘1987’, WMF ‘1993’) 1996Web (PNG ‘1996’, SVG ’2001’..) Vector Graphics Indexing and Retrieval

System overview [Doer’98] [Tom’03] Look like pattern recognition approach Features Extraction Retrieval Index Doc 1 Doc 2 Doc 3 Square Junction Graphics objects Model 1Model 2 Adjacency Line Inclusion Model 3 Content adaptationStructured index Indexed objects Level 2 Level 1 Level Pattern frequency Ranked patterns Our key ideas Features Extraction Retrieval Doc 1 Doc 2 Doc 3 content adaptation structured index Indexing process must adapted to document content We can improve results by structuring the index

Vector Graphics Indexing and Retrieval You see 5 You have 9 Our approach R3 R1 set of objects parsing and break-up set of line filtering then junction detection set of broke line Before retrieve, we need to extract features What are the difficulties ? R3 R1 R2 <rect x="400" y="100" width="400" height="200" fill="blue" /> <rect x="650" y="200" width="400" height="200" fill="yellow" /> How to get R2 ? We need a break-up Sorting the bounding box How to speed up the process ? x 21 x 11 x 12 x 22 y 21 y 11 y 12 y 22 We need a clean- up

Vector Graphics Indexing and Retrieval Our approach (next) line graph building Polyline Junction while 2-connex  edge if  3-connex  node adjacency and inclusion common vectorincluded bounding box Adjacency Polygon Result example gravity center adjacency line inclusion Time processing on ‘Mikado’ database region detection Polygon while  starting vector take nearest vector [Wen’01] To work on graph take time Using vectorial data

Vector Graphics Indexing and Retrieval Features Extraction Retrieval Doc 1 Doc 2 Doc 3 Performance evaluation GT 1 GT 2 GT 3 To work on retrieval engine now ? How to evaluate the retrieval results after ? We must work on performance evaluation before ? How to get the ground truth ? Produce ground truth from existing document take time, we must produce synthetic document. Our key idea Produce true-life document need much knowledge, it is harder to do with a computer We can produce ‘creasy’ but well formed documents, it is sufficient for performance evaluation purposes Synthetic document production Production rules connected 1-connected 2-connected Production rules 0-n 1 1 O-n ‘Creasy’ but well formed drawing

Vector Graphics Indexing and Retrieval Graphical Objects General rules object number document size object choice -probability distribution -rotation and scale range -position constraints -overlapped or not … Domain rules must be connected must be adjacent must be include can include … Noise rules to scale line to broke line to move line … Low Level Primitives I II III (4) To move objects according to domain rules (5) To delete oldest alone objects ‘cycle number’ (6) Adding noise on low level primitives composing objects while Vector Graphics Ground Truth (1) To insert a new object while underhand object number (2) To move other objects if it can’t do (1) (3) To exit if it can’t do (1) and (2), then run (4) and (5) In progress rotate and scalerotate and distort scale and overlap

Vector Graphics Indexing and Retrieval Works done Fast graph building from vector graphics Production of first synthetic documents Works in progress … To produce more complex synthetic documents … To work on model selection … To work on index structuration … About project dot-line 04/05SCSIT Post doc 02/06IRCSET Application A. Winstanley (NCG, Dublin University) 04/06Eureka Meeting eConnector, HP Lab 06/06ANVAR Application informal agreement 11/06EPEIRES contract 2007To visit A. Winstanley (NCG, Dublin University) To take contact with M. Fonseca (IST, Lisbon University) 2008JM Ogier plan to mount a European project

Plan Short CV Vector Graphics Indexing and Retrieval Dropcap Image Retrieval

Old books of XV° and XVI° centuries Bartolomeo (1534)Alciati (1511)Laurens (1621) figure dropcap headline Book46 Page1385 Graphics4755 (3.4 per page) Foreground pixel [Jour’05] 63% textual 37% graphical Graphics type41% dropcap 59% others Old Graphics Which part and kind of graphics in old books CESR Database

Vascosan 1555Marnef 1576 (1) Wood plug tracking Printing house plug exchange copy Dropcap Image Retrieval In what are interested historian people with these images ? Wood plug (bottom view) Retrieve similar printings Plug 1 Plug 2 Plug 3 Printing 1 Printing 2 Why ? Real time process or not ? We can’t index all images in regard to legal properties, a real time process will allow to do queries with images provided by other digital libraries DB query results result (2) User-driven historical metadata acquisition Metadata file Metadata file Metadata file Metadata file Without retrieval With retrieval more faster reduce error

descriptors fast local complex global To scalar [Loncaric’98]  Hough, Radon, Zernike, Hu, Fourrier  Scaled and orientation invariant  fast  local (character, symbol, digit) To image [Gesu’99]  Template matching, Hausdorff distance  no scaled and orientation invariant  slow  global (scene) Dropcap Image Retrieval Noise Offset Complexity Accuracy Scalability several hundred of classes several thousand of images What are the main difficulties? Which descriptor use ? Not adapted for our problem More adapted but too complex Query Compression Centering and Comparison R1 R2 R3 Filtering Image Database Our key idea To use an image compressed representation

We have started to work with our images but the file formats are so different Dropcap Image Retrieval Compression Centering and Comparison Filtering

Digitalization problems [Lawrence’00] Several image providers Several digitalization tools Long process Human supervised Complex post-processing plate-form … Why ? Dropcap Image Retrieval Compression Centering and Comparison Filtering Contrôle Before to work on retrieval engine historian people need tools to improve quality of their databases Our key idea To develop an engine (QUEID) working on image metadata to detect digitalization problem, and to secure retrieve system Diagnostic Base Expertise QUEID query charts analysis Format Diagnostic mode (1)Software setting (2)Image exchange (3)Prototype software 250 to 350Resolutions UncompressCompression TiffFormats grayModel MpSize 2038Files QUEID Engine Base accepted rejected Parameters Filtering mode Our database

Dropcap Image Retrieval To use a Run Length Encoding (RLE) of Image Our key idea image foreground background both Which kind of RLE ? both RLE seems more adapted Compression results Compression Centering and Comparison Filtering

Centering x2x2 x2x2 x2x2 x1x1 x1x1 x1x1 x2x2 x2x2 x1x1 line (y) image 1 line (y+d y ) image 2 x stack reference while x 2  x 1 handle image 2 while x 1  x 2 handle image 1 Comparison Time results Raster vs RLE Max Mean Min Time s Size k.pixel Max Mean Min Time s Size k.run image database query image Dropcap Image Retrieval Compression Centering and Comparison Filtering To solve the offset problems we must use a centering step before the comparison We can do it in an easy way by comparing foreground histogram

Mean query of 40 s, how to reduce again without using a lossless compression and to loose accuracy ? Level 1 : image sizes Level 2 : black, white pixels Level 3 : RLE comparison Our first system Dropcap Image Retrieval How to process the distance curve ? 11 22 if  1 -  2 < 0 push x, cluster while  1 -  2 < 0 next Using a basic clustering algorithm ‘elbow criteria’ query 1 st Level 2 sd Level To use a system appraoch using different level of operator (from more speed to more accurate) to select image to compare Our key idea Speed Depth

Selection results 59%Max 24%Mean 4%Min Selection % From 4% to 59%, how to reduce the variability ? To work on a better selection criteria seems ambiguous … Dropcap Image Retrieval To add an intermediate operator between scalar and image data Our key idea

Base IHM Retrieve engine control display retrieve Labels driven labelling Bench1Bench2 To produce Example of query result Same plug Next plug Query First results seem good, but how to get the ground truth and to evaluate our system? Dropcap Image Retrieval To use our engine to produce benchmark database Our key idea

Works done QUEID to filter and analyse image database Speedup comparison using two feature RLE compression System approach Works in progress … To add operator to improve system To extend our system to produce benchmark database About project dot-line 09/05MADONNE Postdoc 06/061er CESR Technical Meeting 09/06ANAGRAM Worshop (Fribourg) 10/062sd CESR Technical Meeting 10/06NaviDoMass agreement 2007GDR-JC Project (LMA, LI, CreSTIC, LITIS, CVC) To put online the system on CESR website old graphic working group (Glasgow, Tours …) Dropcap Image Retrieval

Bibliography 1.J. Mong and D. Brailsford. Using svg as the rendering model for structured and graphically complex web material. In Symposium on Document Engineering (DocEng), pages 88-91, Y. Chen, J. Gong, W. Jia, and Q. Zhang. Xml-based spatial data interoperability on the internet. In Conference of International Society for Photogrammetry and Remote Sensing and Spatial Information Sciences (ISPRS), pages , J. Kang, B. Lho, J. Kim, and Y. Kim. Xml-based vector graphics: Application for web-based design automation. In International Conference on Computing in Civil and Building Engineering (ICCCBE), pages , M. Weindorf. Structure based interpretation of unstructured vector maps. In Workshop on Graphics Recognition (GREC), volume 2390 of Lecture Notes in Computer Science (LNCS), pages , N. Journet, R. Mullot, J. Ramel, and V. Eglin. Ancient printed documents indexation: a new approach. In International Conference on Advances in Pattern Recognition (ICAPR), volume 3686 of Lectures Notes in Computer Science (LNCS), pages , V. D. Gesu and V. Starovoitov. Distance based function for image comparison. Pattern Recognition Letters (PRL), 20(2): , S. Loncaric. A survey of shape analysis techniques. Pattern Recognition (PR), 31(8): , G. Lawrence and al. Risk management of digital information: A file format investigation. RLG DigiNews, 8(4), 2000.