Concepts, Semantics and Syntax in E-Discovery David Eichmann Institute for Clinical and Translational Science The University of Iowa David Eichmann Institute for Clinical and Translational Science The University of Iowa

Our Approach  Analyze the human-generated metadata available for document collections for organizational and individual interactions  Explore the syntactic and semantic nature of document content and the potential for automatic generation of metadata  Explore the concept space generated by the previous step and its correspondence to boolean predicate specification in discovery

Our Target Corpus  The Illinois Institute of Technology Complex Document Information Processing Test Collection (IIT CDIP), v. 1.0  Derived from the tobacco master settlement agreement  Comprises 6,910,192 ‘documents’  Or more properly the OCR output from those documents  Two merged XML tag sets of metadata, with overlapping content 

Metadata Entity Frequencies Entity Occurrences TotalDistinctAvg/EntityAvg/Doc. Bates9,476,7948,054, Category13,594, , Doctype18,359,6442,5017, Prodbox6,830,9936,3061,

Metadata Entity Frequencies Entity Occurrences TotalDistinctAvg/EntityAvg/Doc. Attendee65,691,47349,3751, Brand26,498,001155, Copied8,775,307322,

Metadata Entity Frequencies Org. Entity Occurrences TotalDistinctAvg/EntityAvg/Doc. Author8,742,976149, Mentioned31,406,753883, Receiving8,262,49663,

Metadata Entity Frequencies Person Entity Occurrences TotalDistinctAvg/EntityAvg/Doc. Author11,128,029875, Mentioned34,683,2891,938, Receiving23,427,415455,

Database Schema  We map the XML structure to a set of relational database tables  Non-recurring fields are collected in a table named ‘document’  docid  title  description  OCR text  Recurring elements each get a table  docid  value

Identifying an Individual Person # of Occurrences as AttendeeAuthorReceiverMention REININGHAUS, W189,38023,88032,76416,152 REININGHAUS7, ,9742,837 REININGHAUS, B1962 REININGHAUS, R

How Many Reininghaus?  Reininghaus,R  Reininghaus,W

Co-mention Connections ReininghausWalk PersonCountPersonCount WALK,RA3,871REININGHAUS,W3,871 ROEMER,E3,716ROEMER,E2,883 HAUSSMANN,HJ3,293HAUSSMANN,HJ2,799 TEWES,F2,784HACKENBERG,U2,360

Co-mention Connections ReininghausRoemer PersonCountPersonCount WALK,RA3,871REININGHAUS,W3,716 ROEMER,E3,716WALK,RA2,883 HAUSSMANN,HJ3,293HACKENBERG,U2,623 TEWES,F2,784HAUSSMAN,HJ2,573

Co-mention Connections ReininghausHaussmann PersonCountPersonCount WALK,RA3,871REININGHAUS,W3,293 ROEMER,E3,716WALK,RA2,799 HAUSSMANN,HJ3,293ROEMER,E2,573 TEWES,F2,784VONCKEN,P2,323

Co-mention Affiliations PersonAffiliation Reininghaus, Wolf Gen. Mgr, Contract Research, INBIFO Walk, Rudiger-AlexanderDir. Human Studies, Philip Morris Roemer, EwaldINBIFO Haussmann, Hans-JurgenAssoc. Prin. Scientist, Philip Morris Tewes, F.Biologist, INBIFO Hackenberg, UlrichINBIFO Voncken, P.Chemist, INBIFO

Semantics and Structure  Our analysis of content involves the following phases:  Lexical analysis  Sentence boundary detection  Named entity recognition  Sentence parsing  Relationship extraction  The nature of the OCR data seriously impacts each of the phases (sometimes in different ways)

CDIP Parse Tree Complexity

Clean Text Parse Tree Complexity

Next Steps  Experiment with custom lexical analysis of the OCR  Start with simple white space detection  Construct a lexicon and look for out-of-band vocabulary as OCR errors candidates  Rewrite the analyzer to support OCR error correction  Sentence boundary detect and parse the full corpus  Generate entity relationships using our question answering framework

And Beyond That…  Return to the document images and analyze document layout  Regenerate OCR to include token coordinates  Use our PDF structure extraction framework to generate logical document structure  Generate a set of document models based upon similar layout  Use the document models to map OCR text to metadata elements

For Example