1. Logic Programming for Natural Language Processing Menyoung Lee TJHSST Computer Systems Lab Mentor: Matt Parker Analytic Services, Inc.

2. Purpose To link together  Recent developments in natural language processing (NLP): Information Extraction (IE)  Classical logic programming: Prolog New Paradigm: bifurcated process  An IE application which will produced structured output from a corpus of free, unstructured text.  Transformation of extracted information into a Prolog knowledge-base (sets of fact-triples) Documents: biographies

3. Why NLP? Language is the cornerstone of intelligence  The Turing Test: the ability to converse like man Understanding and generating texts in a natural language, e.g. English Many specific NLP tasks  Chatterbots, e.g. Eliza  Machine Translation  Information Retrieval (IR), e.g. Google  Information Extraction!!  SciFi Dreams: universal translation, computers you can talk to, etc.

4. Information Extraction (IE) Most generally, the transformation of  Information contained in free, unstructured text in a natural language into  A prescribed, structured format. More specifically, the identification of  Instances of certain object classes  Their attributes  Relationships between object instances Always restricted into a particular domain  In order to have a reasonably sized and sufficiently expressive ontology

5. Why IE? An Expert must read many documents Advent of the Internet & Information Age  Explosion of the sheer volume of textual information, readily available in electronic form  New opportunity: lots and lots of available information to exploit  Formidable challenge: impossible for an expert to read and analyze that much text. A pragmatic approach:  Full text understanding is out of reach  Automate just some of the tasks, i.e. the identification of objects, attributes, and relations

6. IE - Details Five Tasks in IE  Named Entity Recognition (NE)  Coreference Resolution (CO)  Template Element Construction (TE)  Template Relation Construction (TR)  Scenario Template Production (ST) Metrics for Evaluation  Precision:  Recall:  F-measure (borrowed from IR): More intuitive reformulation:

7. Annotations Annotations identify objects in text Annotation graph: a directed, acyclic graph (DAG)  Nodes position in the text  Edges The literal text Annotations

8. Frames Frame: representation of an object, consisting of slots, which contain values Typical Prolog fact: Frame(Slot, Value). We propose to synthesize it with the idea of annotations: Doc(Annot, Text).  Main idea: represent the document directly as an object: compromise between text and knowledge Several Advantages  A corpus of multiple related documents  Direct link between information and its source  Opens the door for the application of Prolog's logic.

9. Design The IE application  Input: corpus of free, unstructured text  Output: the annotated documents, represented as annotation graphs  How: use GATE (language: JAPE) The Prolog application  Input: the annotated document  Output: a frame, i.e. a set of Prolog facts.  How: use XSB (language: Prolog)

10. General Architecture for Text Engineering (GATE) A comprehensive architecture for development of NLP applications Documents treated as an annotation graph Java Annotation Patterns Engine  Its own language for writing grammars that identify instances of object classes to annotate A Nearly New Information Extraction (ANNIE) system  An already implemented rudimentary IE system, that can be extended through addition of JAPE grammars for annotating Machine-learning models for annotating

11. GATE

12. Procedures Obtain the corpus – Python script Write the Jape grammars  annotations 'Mathematician', 'Father'. Train a model  annotation 'Protagonist' Write the Prolog application to  Parse GATE's XML output into a structure  Construct the annotation graph from it  Process the annotations into a document frame  Output the document frame Test by posing queries

13. IE Result: Fermat.html Precision: 1. (why so high?)  use of a gazetteer list  aggressive pruning by context Recall: 0.9474  paid for aggressive pruning, missed some F-measure (β = 2)  0.973

14. Prolog Result Correctly constructs facts. Sample session: | ?- 'Galois.html.xml'('Mathematician', X). X = Abel; X = Cauchy; X = Evariste Galois; X = Fourier; X = Galois; X = Gauss; X = Gergonne; X = Jacobi; X = Lagrange; X = Legendre; X = Libri; X = Liouville; X = Poisson; X = Vernier

15. Results The Prolog layer is universal, cross-domain  The IE application may produce any annotation, not restricted to one subject area  Bifurcation: success Opens door to logic and rules, esp. for cross- document relations | ?- 'Galois.html.xml'('Mathematician', X), 'Cauchy.html.xml'('Protagonist', X). X = Cauchy; no

16. Conclusion With the recent advancements in computing power, logic programming is finally feasible for practical use  To run my Prolog application, ran it on the server robustus, giving it 2 GB of memory  However, computing power continues to be a limitation (GATE crashed every day) Where do we go from here?  More expressive document frame  Context analysis (through proximity, etc)  Better IE applications through statistical processing