Dept. of Computer Science University of Liverpool COMP527: Data Mining COMP527: Data Mining M. Sulaiman Khan (mskhan@liv.ac.uk)‏ Dept. of Computer Science University of Liverpool 2009 Text Mining: Text-as-Language March 26, 2009 Slide 1

COMP527: Data Mining COMP527: Data Mining Introduction to the Course Introduction to Data Mining Introduction to Text Mining General Data Mining Issues Data Warehousing Classification: Challenges, Basics Classification: Rules Classification: Trees Classification: Trees 2 Classification: Bayes Classification: Neural Networks Classification: SVM Classification: Evaluation Classification: Evaluation 2 Regression, Prediction Input Preprocessing Attribute Selection Association Rule Mining ARM: A Priori and Data Structures ARM: Improvements ARM: Advanced Techniques Clustering: Challenges, Basics Clustering: Improvements Clustering: Advanced Algorithms Hybrid Approaches Graph Mining, Web Mining Text Mining: Challenges, Basics Text Mining: Text-as-Data Text Mining: Text-as-Language Revision for Exam Text Mining: Text-as-Language March 26, 2009 Slide 2

Ontologies, Entity Extraction Today's Topics COMP527: Data Mining Part of Speech Tagging Deeper Parsing Ontologies, Entity Extraction Text Mining: Text-as-Language March 26, 2009 Slide 3

This is often represented as: word/XYZ Part of Speech Tagging COMP527: Data Mining The first step is to tag each word with additional information about it. The first bit of information is the word's part of speech. This is often represented as: word/XYZ Where XYZ are some uppercase characters standing for the part of speech of that particular word. There isn't a standard for these tags, but the Penn Treebank set is commonly used, sometimes with extensions. Text Mining: Text-as-Language March 26, 2009 Slide 4

Noun: Naming word (cat, dog, Rob, television)‏ Part of Speech Tagging COMP527: Data Mining Part of Speech Tagging aims to find the part of speech of a word – eg noun, verb, adjective, adverb, conjunction, determiner, pronoun, preposition. Noun: Naming word (cat, dog, Rob, television)‏ Verb: Action word (jump, fish, sell, watch, say)‏ Adjective: Qualifier for noun (big, red, expensive, hungry)‏ Adverb: Qualifier for verb (quickly, noisily, efficiently)‏ These are 'content' words (and hence ones to keep!)‏ Text Mining: Text-as-Language March 26, 2009 Slide 5

Other four main types of part of speech: Part of Speech Tagging COMP527: Data Mining Other four main types of part of speech: Conjunction: Joining words (and, hence, then)‏ Determiner: Articles (the, a, an)‏ Pronoun: Word that stands for a noun (he, it, them, her) Preposition: 'Direction' words (to, with, in, of) These are 'function' words (and hence ones to throw away!)‏ But PoS taggers have a much larger set of classes. Text Mining: Text-as-Language March 26, 2009 Slide 6

Common Penn Treebank Tags: Part of Speech Tagging COMP527: Data Mining Common Penn Treebank Tags: Text Mining: Text-as-Language March 26, 2009 Slide 7

“This is a crazy sentence written by Rob” Examples COMP527: Data Mining “This is a crazy sentence written by Rob” this/DT is/VBZ a/DT crazy/JJ sentence/NN written/VBN by/IN Rob/NNP “Where have all the flowers gone?” Where/WRB have/VBP all/PDT the/DT flowers/NNS gone/VBN ?/. “The cat sat on my house's welcome mat” The/DT cat/NN sat/VBD on/IN my/PRP$ house/NN 's/POS welcome/JJ mat/NN Text Mining: Text-as-Language March 26, 2009 Slide 8

Gets 95% accuracy using about 70 rules Taggers COMP527: Data Mining 90% of words only have one part of speech. So a simple dictionary lookup will get ~90% accuracy, then need a model to predict the other words. Of the words that have more than one PoS, about ½ are noun/verb, and >80% are noun/verb or noun/adjective. Could build a decision tree tagger using this information, with some simple additional rules for non dictionary words and words with multiple PoS. Brill did this, with some additional training methods to correct errors. Gets 95% accuracy using about 70 rules Text Mining: Text-as-Language March 26, 2009 Slide 9

Word Sense Disambiguation COMP527: Data Mining Just because we know the PoS, doesn't mean we know what the word means: The state has lowered its tax rate. The company is in a weak financial state. The solid state of water is ice. Can you state your opinion? (yes this use is a verb)‏ Eg: region, condition, state of matter, make a statement. Can use Latent Semantic Indexing/Clustering techniques to implement WSD. (eg financial + state vs solid/gas/liquid +state)‏ Text Mining: Text-as-Language March 26, 2009 Slide 10

Word Sense Disambiguation COMP527: Data Mining WordNet is a very commonly used 'dictionary' from Princeton. Includes links to synonyms (called synsets): [dog, canine, puppy, hound, ...] and higher/lower semantic categories: dog – canine – carnivore – mammal – animal – living thing Can use to help disambiguate word senses and link words together. Text Mining: Text-as-Language March 26, 2009 Slide 11

That's all very nice, but ... so what? Deep Parsing COMP527: Data Mining That's all very nice, but ... so what? For Data Mining, knowing the part of speech for a word could be useful, but it's just the first step in Text Mining. Step 2 is to extend the parsing to find phrases and how the verbs and nouns match up. Often called 'chunks', these systems find sequences of the words based on the parts of speech. Text Mining: Text-as-Language March 26, 2009 Slide 12

(NP (DT the) (NN cat) ) // noun phrase Deep Parsing COMP527: Data Mining (TOP (S (NP (DT the) (NN cat) ) // noun phrase (VP (VBD sat) // verb phrase start (PP (IN on) (NP // second noun phrase (NP (PRP$ my) (NN house) (POS 's) ) // sub phrase (JJ welcome) (NN mat)‏ )‏ ))‏ Text Mining: Text-as-Language March 26, 2009 Slide 13

So arg1 of bites is dog, and arg2 is man. Dog Bites Man. Deep Parsing COMP527: Data Mining We can also find out which verbs and which nouns correlate, in which order. EG, does the man bite the dog, or the dog bite the man? “The dog bites the man” ROOT ROOT ROOT ROOT ROOT bites bites bite VBZ VB ARG1 dog bites bite VBZ VB ARG2 man The the DT DT ARG1 dog the the DT DT ARG1 man So arg1 of bites is dog, and arg2 is man. Dog Bites Man. What about if we mix it up a little... Text Mining: Text-as-Language March 26, 2009 Slide 14

“The man was bitten by the dog” Deep Parsing COMP527: Data Mining “The man was bitten by the dog” ROOT ROOT ROOT ROOT ROOT bitten bitten bite VBN VB ARG1 dog bitten bite VBN VB ARG2 man The the DT DT ARG1 man the the DT DT ARG1 dog by by IN IN ARG1 dog was be VBD VB ARG1 man was be VBD VB ARG2 bitten So arg1 of bites is still dog, and arg2 is still man. Dog still Bites Man, even with the language mixed up a bit. Text Mining: Text-as-Language March 26, 2009 Slide 15

Ontologies COMP527: Data Mining We have a Proper Noun indicator (NNP). Proper nouns name a specific person, place or thing. These may have some unique identifier, or other associated information. It may exist in an ontology (or database with a similar purpose). Ontologies are classification systems that describe objects and the relationships between them. For example, a directory of people could be an ontology. With some hints as to the type of object, we can look it up in these databases to find out extra information about it in an easily machine- understandable format. Text Mining: Text-as-Language March 26, 2009 Slide 16

A common ontology meta-language is OWL (Web Ontology Language). Ontologies COMP527: Data Mining At this point we have most of the picture in place. We can identify entities and find out additional information, and know which entities act in which way on other entities. A common ontology meta-language is OWL (Web Ontology Language). It in turn uses RDF (Resource Description Framework). RDF's main advantages are allowing inference engines to traverse a large graph of objects and relations, which is the next step in generating correlations between objects and relationships extracted from the text. (See also Semantic Web etc.)‏ Text Mining: Text-as-Language March 26, 2009 Slide 17

Named Entity Recognition COMP527: Data Mining Find 'entities' to extract from text, typically to then look up in an ontology. Eg: "City University of New York" vs "John Smith of New York". Can list countries, even major cities, but how to determine which 'Springfield' is being discussed? Need context, etc. Similar sorts of problems to PoS tagging in terms of sequence analysis, and same sorts of solutions. Classes might be: person, organisation, place, time, date, not-an-entity. Can use HMM, rule based, etc. Text Mining: Text-as-Language March 26, 2009 Slide 18

Events COMP527: Data Mining Temporal events described in text are also important to recognise. For example 'the dog bites the man' is a one off event that happens at a particular time. It might be useful to be see that the man then went to hospital after being bitten. Event extraction can be done via rules, which will be specific to a domain: Templates of different noun types and verbs in given orders that are important to recognise. For example, a person getting a new job title at a company is indicative of a promotion or new acquisition. Text Mining: Text-as-Language March 26, 2009 Slide 19

(How nice would that be!)‏ (Though it is being worked on)‏ Summarisation COMP527: Data Mining If the system can extract events, entities, etc etc, then the 'next' step is to summarise the findings from across the various bits of information extracted. (How nice would that be!)‏ (Though it is being worked on)‏ Text Mining: Text-as-Language March 26, 2009 Slide 20

Anything on Natural Language Processing Further Reading COMP527: Data Mining Konchady Weiss Anything on Natural Language Processing Text Mining: Text-as-Language March 26, 2009 Slide 21