Text Analytics / NLP Charan Puvvala, Mirabel Technologies Inc. Hyderabad Machine Learning Group

Outline Introduction Natural Language Processing Linguistic Ambiguities Typical NLP tasks Information Extraction Named Entity Recognition

80:20 - Pareto Principle The principle was suggested by management thinker Joseph M. Juran. It was named after the Italian economist Vilfredo Pareto, who observed that 80% of income in Italy was received by 20% of the Italian population. The assumption is that most of the results in any situation are determined by a small number of causes. Different Twist: 80% of the features are done in 20% of the time and 20% of the laborious features take 80% of the time.

Introduction - NLP definition Natural Language Processing Natural (adjective) - existing in nature and not made of cause by people: coming from nature. Merriam-Webster dictionary Language (noun) - the system of words and sounds that people use to express thoughts and feeling to each other. Merriam-Webster dictionary Process (verb) - to subject or handle through an established usually a routine set of procedures. Merriam-Webster dictionary

Technology making good progress still really hard mostly solved ✓ ✗ Sentiment analysis still really hard mostly solved Best roast chicken in Hyderabad! Question answering (QA) The waiter ignored us for 20 minutes. Q. How effective is ibuprofen in reducing fever in patients with acute febrile illness? Spam detection Coreference resolution ✓ Let’s go to Agra! “I voted for Charan because he was most aligned with my values” he said ✗ Paraphrase Buy Football… Word sense disambiguation (WSD) XYZ acquired ABC yesterday Part-of-speech (POS) tagging I need new batteries for my mouse. ABC has been taken over by XYZ ADJ ADJ NOUN VERB ADV Summarization Parsing Colorless green ideas sleep furiously. The Dow Jones is up Economy is good I can see Alcatraz from the window! The S&P500 jumped Named entity recognition (NER) Housing prices rose Machine translation (MT) PERSON ORG LOC Dialog 第13届上海国际电影节开幕… Where is Citizen Kane playing in NY? Einstein met with UN officials in Princeton The 13th Shanghai International Film Festival… IMax Theatre at 7:30. Do you want a ticket? Information extraction (IE) Party May 27 add You’re invited to our dinner party, Friday May 27 at 8:30

Introduction Dave Bowman: Open the pod bay doors, HAL. HAL: I’m sorry Dave, I’m afraid I can’t do that. (Stanley Kubrick & Arthur C. Clarke, screenplay of 2001: A Space Odyssey )

Introduction - Basic concepts Syntax Open the pod bay doors, HAL. Vs. HAL, is the pod bay door open? Lexical Semantics meaning of component words Compositional Semantics knowledge of how components combine to form large meanings - Colorless green ideas sleep furiously. <-> Dark green leaves rustle furiously. Pragmatics I’m sorry ... , I’m afraid I can’t Vs. No, I won’t open the door. Phonetics & Phonology Morphology Produce contractions I’m, can’t Discourse conventions engaging in structured conversation using reference that I’m sorry Dave, I’m afraid I can’t do that syntax is about sentence formation, and semantics about sentence interpretation, phonetics and phonology cover the field of sentence utterance.

Maths Computer Science Linguistics NLP

Ambiguities Linguistic Ambiguities Syntactic Ambiguities Morphological Ambiguities Syntactic Ambiguities Semantic Ambiguities

Linguistic Ambiguities Example: I made her duck Five possible interpretations I cooked waterfowl/duck for her I cooked waterfowl belonging to her I created the (plaster?) duck she owns I caused to quickly lower her head or body I waved my magic wand and turned her into waterfowl

Linguistic Ambiguities Morphological ambiguity duck: verb or noun her: dative pronoun or possessive pronoun Syntactic ambiguity: make transitive: taking a single direct object (case 2) ditransitive: taking two objects, meaning the first object (her) got made into the second object (duck) taking a direct object and verb, meaning that the object (her) got caused to perform the verbal action (duck) Semantic ambiguity: make cook create

Typical NLP Tasks - Simpler Tokenization RegEx Sentence Splitting RegEx POS-tagging algorithms and tag sets POS-Tagging

Typical NLP tasks - Complex Lemmatization / Stemming Syntactic Parsing Question Answering systems Topic Extraction NER Semantic Analysis Knowledge Graphs

semi-structured information Unstructured information How to link this information to a knowledge base automatically?

Information Extraction Information Extraction (IE) systems Find and understand limited relevant parts of text Gather information from many pieces of text Produce a structured representation of relevant information: Relations (in the database sense) A knowledge base Goals: Organize information so that it is useful to people Put information in a semantically precise form that allows further inferences to be made by computer algorithms Now let's define information extraction! a more general task Goal: get semantic information out of documents, esp. web pages Defined as a dumbing-down of more lofty goal of Natural Language Understanding -- more technologically manageable we're often interested in learning about particular relations (in DB sense) e.g. scouring financial news for movements of executives e.g. [person] [assumes/loses] [role] at [company] I want to scour through text, find relation instances, suck out, put in DB you're allowed to do domain- and problem-specific customization lots of potential applications - business/financial context - biomedical context, clinical medicine there's all this unstructured text data about research and patients -- you'd like to be able to get structured information out of it could lead to other automated information finding: trends, correlations, drug interactions, impact of some protein on expression of a gene, ...

Information Extraction IE systems extract clear, factual information Roughly: Who did what to whom when? E.g., Gathering earnings, profits, board members, headquarters, etc. from company reports The headquarters of BHP Billiton Limited, and the global headquarters of the combined BHP Billiton Group, are located in Melbourne, Australia. headquarters(“BHP Biliton Limited”, “Melbourne, Australia”) Learn drug-gene product interactions from medical research literature Now let's define information extraction! a more general task Goal: get semantic information out of documents, esp. web pages Defined as a dumbing-down of more lofty goal of Natural Language Understanding -- more technologically manageable we're often interested in learning about particular relations (in DB sense) e.g. scouring financial news for movements of executives e.g. [person] [assumes/loses] [role] at [company] I want to scour through text, find relation instances, suck out, put in DB you're allowed to do domain- and problem-specific customization lots of potential applications - business/financial context - biomedical context, clinical medicine there's all this unstructured text data about research and patients -- you'd like to be able to get structured information out of it could lead to other automated information finding: trends, correlations, drug interactions, impact of some protein on expression of a gene, ...

Information Extraction Between AD 1400 and 1450, China was a global superpower run by one family – the Ming dynasty – who established Beijing as the capital and built the Forbidden City.

Named Entities Between AD 1400 and 1450, China was a global superpower run by one family – the Ming dynasty – who established Beijing as the capital and built the Forbidden City. Name Entity Recognition (NER)

Named Entities Between AD 1400 and 1450, China was a global superpower run by one family – the Ming dynasty – who established Beijing as the capital and built the Forbidden City. Name Entity Recognition (NER) China: /location/country Ming dynasty: /royalty/royal_line Beijing: location/city Forbidden city: /location/city

Named Entities: Definition Named Entities: Proper nouns, which refer to entities Named Entity Recognition: Detecting boundaries of named entities (NEs) Named Entity Classification: Assigning classes to NEs, such as PERSON, LOCATION, ORGANISATION, or fine-grained classes such as ROYAL LINE Named Entity Linking / Disambiguation: Linking NE’s to concrete entities in knowledge base China -> Location: Republic of Chine, country in East Asia, Person -> Person: China, Brazilian footballer born in 1964 Music -> China, a 1979 album by Vangelis

Information Extraction: Methods Possible methodologies Rule-based approaches: write manual extraction rules Machine learning based approaches Supervised learning: manually annotate text, train machine learning model Unsupervised learning: extract language patterns, cluster similar ones Semi-supervised learning: start with a small number of language patterns, iteratively learn more (bootstrapping) Gazetteer-based method: use existing list of named entities Combination of the above

NLP & ML Software Natural Language Processing: - GATE (general purpose architecture, includes other NLP and ML software as plugins) - Stanford NLP (Java) - OpenNLP (Java) - NLTK (Python) Machine Learning: - scikit-learn (Python, rich documentation, highly recommended!) - Mallet (Java) - WEKA (Java) - Alchemy (graphical models, Java) - FACTORIE (graphical models, Scala) - CRFSuite (efficient implementation of CRFs, Python)

NLP & ML Software Ready to use NERC software: - ANNIE (rule-based, part of GATE) - Wikifier (based on Wikipedia) - FIGER (based on Wikipedia, fine-grained Freebase NE classes) Almost ready to use NERC software: - CRFSuite (already includes Python implementation for feature extraction, you just need to feed it with training data, which you can also download) Ready to use RE software: - ReVerb (Open IE, extracts patterns for any kind of relation) - MultiR (Distant supervision, relation extractor trained on Freebase) Web Content Extraction software: - Boilerpipe (extract main text content from Web pages) - Jsoup (traverse elements of Web pages individually, also allows to extract text)