1. The ODU Metadata Extraction Project March 28, 2007 Dr. Steven J. Zeil zeil@cs.odu.edu

2. Outline 1.Overview 2.Recent Developments A.Independent Document Model B.Validation C.Diversifying – NASA & GPO collections 3.New Issues & Future Directions A.Post-processing B.Image-Based Classification

3. 1. Overview

4. Input Processing & OCR Select pages of interest Apply Off-The-Shelf OCR software Convert OCR output to XML model format

5. Form Processing Scan document for form names –Select form template Apply form extraction engine to document and template

6. Sample RDP

7. Sample RDP (cont.)

8. Metadata Extracted from Sample RDP (1/3) 18-09-2003 Final Report 1 April 1996 - 31 August 2003 VALIDATION OF IONOSPHERIC MODELS F19628-96-C-0039 61102F Patricia H. Doherty Leo F. McNamara Susan H. Delay Neil J. Grossbard 1010 IM AC Boston College / Institute for Scientific Research 140 Commonwealth Avenue Chestnut Hill, MA 02467-3862

9. Metadata Extracted from Sample RDP (2/3) Air Force Research Laboratory 29 Randolph Road Hanscom AFB, MA 01731-3010 VSBP AFRL-VS-TR-2003-1610 Approved for public release; distribution unlimited. This document represents the final report for work performed under the Boston College contract F I9628-96C-0039. This contract was entitled Validation of Ionospheric Models. The objective of this contract was to obtain satellite and ground-based ionospheric measurements from a wide range of geographic locations and to utilize the resulting databases to validate the theoretical ionospheric models that are the basis of the Parameterized Real-time Ionospheric Specification Model (PRISM) and the Ionospheric Forecast Model (IFM). Thus our various efforts can be categorized as either observational databases or modeling studies.

10. Metadata Extracted from Sample RDP (3/3) Ionosphere, Total Electron Content (TEC), Scintillation, Electron density, Parameterized Real-time Ionospheric Specification Model (PRISM), Ionospheric Forecast Model (IFM), Paramaterized Ionosphere Model (PIM), Global Positioning System (GPS) John Retterer 781-377-3891 U U SAR

11. Non-Form Processing Classification – compare document against known document layouts –Select template written for closest matching layout Apply non-form extraction engine to document and template

12. Non-Form Sample (1/2)

13. Non-Form Sample (2/2)

14. Template Used for Sample Document AU/ onesection AIR COMMAND | AIR WAR AIR UNIVERSITY CorporateAuthor by …

15. Metadata Extracted From the Title Page of the Sample Document AU/ACSC/012/1999-04 AIR COMMAND AND STAFF COLLEGE AIR UNIVERSITY INTEGRATING COMMERCIAL ELECTRONIC EQUIPMENT TO IMPROVE MILITARY CAPABILITIES Jeffrey A. Bohler LCDR, USN Advisor: CDR Albert L. St.Clair April 1999

16. Post-Processing Coerce extracted values into standard formats

17. Validation Estimate quality of extracted metadata Untrusted outputs referred (to humans) for review and correction

18. Recent Developments A.Independent Document Model B.Validation C.Diversifying – NASA and GPO Collections

19. A. Independent Document Model (IDM) Platform independent Document Model Motivation –Dramatic XML Schema Change between Omnipage 14 and 15 –Tie the template engine to stable specification –Protects from linking directly to specific OCR product –Allows us to include statistics for enhanced feature usage Statistics (i.e. avgDocFontSize, avgPageFontSize, wordCount, avgDocWordCount, etc..)

20. Documents in IDM A document consists of pages pages are divided into regions regions may be divided into –blocks of vertical whitespace –paragraphs –tables –images paragraphs are divided into lines lines are divided into words All of these carry standard attributes for size, position, font, etc.

21. Generating IDM Use XSLT 2.0 stylesheets to transform –Supporting new OCR schema only requires generation of new XSLT stylesheet. -- Engine does not change

22. IDM Usage OmniPage 14 XML Doc OmniPage 15 XML Doc Other OCR Output XML Doc IDM XML Doc Form Based Extraction Non Form Extraction docTreeModelOther.xsl docTreeModelOmni15.xsl docTreeModelOmni14.xsl

23. IDM Tool Status Converters completed to generate IDM from Omnipage 14 and 15 XML –Omnipage 15 proved to have numerous errors in its representation of an OCR’d document –Consequently, not recommended Form-based extraction engine revised to work from IDM Non-form engine still works from our older “CleanXML” –convertor from IDM to CleanXML completed as stop-gap measure –direct use of IDM deferred pending review of other engine modifications

24. B. Validation Given a set of extracted metadata –mark each field with a confidence value indicating how trustworthy the extracted value is –mark the set with a composite confidence score Fields and Sets with low confidence scores may be referred for additional processing –automated post-processing –human intervention and correction

25. Validating Extracted Metadata Techniques must be independent of the extraction method A validation specification is written for each collection, combining Field-specific validation rules –statistical models derived for each field of text length % of words from English dictionary % of phrases from knowledge base prepared for that field –pattern matching

26. Sample Validation Specification Combines results from multiple fields <val:validate collection="dtic" xmlns:val="jelly:edu.odu.cs.dtic.validation.ValidationTagLibrary" >...

27. Validation Spec: Field Tests Each field is subjected to one or more tests …...

28. Sample Input Metadata Set Thesis Title: The Military Extraterritorial Jurisdiction Act Name of Candidate: LCDR Kathleen A. Kerrigan Accepted this 18th day of June 2004 by:

29. Sample Validator Output Thesis Title: The Military Extraterritorial Jurisdiction Act Name of Candidate: LCDR Kathleen A. Kerrigan Accepted this 18th day of June 2004 by:

30. Classification (a priori) Previously, we had attempted various schemes for a priori classification –x-y trees –bin classification Still investigating some –image-based recognition

31. Post-Hoc Classification Apply all templates to document –results in multiple candidate sets of metadata Score each candidate using the validator –Select the best-scoring set

32. Experimental Results Manually Assigned Class Number of Documents Validator PreferredTotal Au86000 Eagle0833445 Rand008412 Title0012324

33. Interpretation of Results Validator agreed with human on 125 out of 167 cases Of 42 cases where they disagreed –37 were due to “extra” words in extracted metadata (e.g., military ranks in author names) highlights need for post-processing to clean up metadata –2 were mistakes by template –2 were due to garbled characters by OCR –1 due to a bug in the validator

34. C. Diversifying – NASA and GPO Collections Document collections differ in whether forms are used and form layout document layout what metadata fields are present & which ones are collected

35. Changing Collections Porting to a new document collection –identify pages of interest –training classifiers to recognize new document layouts (?) –templates for forms & document layouts –new validation scripts collect statistics for collection model –new post-processing rules No changes required to core engines & other software

36. NASA Technical Reports Different layouts than DTIC –fewer total –tend to be visually more similar –mixture with and without RDPs

37. NASA Sample Document

38. Extracted Metadata for NASA Sample A Computationally Efficient Meshless Local Petrov-Galerkin Method for Axisymmetric Problems I.S. Raju* and T. Chen? NASA Langley Research Center Hampton, VA 23681 The Meshless Local Petrov-Galerkin (MLPG) method is one of the recently developed element-free …

39. Govt. Printing Office Congressional acts & reports EPA reports Preliminary study with Acts of Congress and EPA reports samples suggest layouts are more diverse than DTIC or NASA –metadata actually present in document varies widely

40. GPO Sample – Act of Congress

41. Metadata Extracted for Act of Congress 118 STAT. 3984 PUBLIC LAW 108?493?DEC. 23, 2004 [H.R. 5394 ] components. 108th Congress An Act Dec. 23, 2004 To amend the Internal Revenue Code of 1986 to modify the taxation of arrow [H.R. 5394 ] components.

42. GPO sample report

43. Metadata Extracted from GPO Sample Report CHINA?S PROLIFERATION PRACTICES AND ROLE IN THE NORTH KOREA CRISIS HEARING BEFORE THE U.S.-CHINA ECONOMIC AND SECURITY REVIEW COMMISSION ONE HUNDRED NINTH CONGRESS FIRST SESSION MARCH 10, 2005 Printed for the use of the U.S.-China Economic and Security Review Commission Available via the World Wide Web: http://www.uscc.gov

44. 3. New Issues and Future Directions A.Post-Processing B.Image-Based Classification

45. Post-processing WYSIWYG WYG != WYW

46. Post-processing WYSIWYG –What You See is What You Get WYG != WYW

47. Post-processing WYSIWYG –What You See is What You Get WYG != WYW –What You Get is not What You Want

48. Example – DTIC Date Format Document may contain: –March 28, 2007 –3/28/2007 –3/28/07 DTIC requires: –28 MAR 2007

49. Example – Personal Authors

50. Example – Personal Authors (cont.) We extract: Patricia H. Doherty Leo F. McNamara Susan H. Delay Neil J. Grossbard DTIC requires: Patricia H. Doherty ;Leo F. McNamara ;Susan H. Delay ;Neil J. Grossbard NASA requires Patricia H. Doherty Leo F. McNamara Susan H. Delay Neil J. Grossbard

51. Post-Processing Requirements Post-processing rules must vary by –metadata field –collection

52. Post-Processing Architecture

53. Image-Based Classification filter to find likely candidates for validator-based selection of template Looking at a variety of techaniques inspired by work in image recognition

54. Example: Image-Based Classification Example: represent a page using various colors to denote images, text, bold text, etc. find visually most similar pages in documents of known classes –“vote” based on 5 most similar documents

55. Visual Matching Example (1/2)

56. Visual Matching Example (2/2)

57. Conclusions Automated metadata extraction can be performed effectively on a wide variety of documents –Coping with heterogeneous collections is a major challenge Much attention must be paid to “support” issues –validation, post-processing, etc.