Download presentation
Presentation is loading. Please wait.
1
Knowledge and Information Retrieval
Session 8 Data Mining and Information Extraction and XML
2
Agenda Data Mining and Network Mining XML
Information Extraction Architecture Bina Nusantara University
3
Data Mining Data mining (the analysis step of the "Knowledge Discovery in Databases“), is the computational process of discovering patterns in large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics, and database systems. Loadingdata.py: import urllib2 from pylab import plot, show from numpy import genfromtxt, zeros url = ' u = urllib2.urlopen(url) localFile = open('iris.csv','w') localFile.write(u.read()) localFile.close() data = genfromtxt('iris.csv',delimiter=',',usecols=(0,1,2,3)) target = genfromtxt('iris.csv',delimiter=',',usecols=(4),dtype=str) plot(data[target=='setosa',0],data[target=='setosa',2],'bo') plot(data[target=='versicolor',0],data[target=='versicolor',2],'ro') plot(data[target=='virginica',0],data[target=='virginica',2],'go') show()
4
Regression Linear regression is an approach for modeling the relationship between a variable y and one or more variables (or independent variable) denoted X. Regresi.py: from pylab import figure, subplot,plot, hist, xlim, show from numpy.random import rand x = rand(40,1) # explanatory variable y = x*x*x+rand(40,1)/5 # depentend variable from sklearn.linear_model import LinearRegression linreg = LinearRegression() linreg.fit(x,y) from numpy import linspace, matrix xx = linspace(0,1,40) plot(x,y,'o',xx,linreg.predict(matrix(xx).T),'--r') show()
5
Network Mining Often, the data that we have to analyze is structured in the form of networks, for example our data could describe the friendships between a group of facebook users or the coauthor ships of papers between scientists. import networkx as nx G = nx.read_gml('lesmiserables.gml',relabel=True) nx.draw(G,node_size=0,edge_color=’b’,alpha=.2,font_size=7)
6
Sec. 3.1 Cliques It is also interesting to study the network through the identification of its cliques. A clique is a group where a node is connected to all the other ones and a maximal clique is a clique that is not a subset of any other clique in the network. print max(cliques) [u'Joly', u'Gavroche', u'Bahorel', u'Enjolras', u'Courfeyrac', u'Bossuet', u'Combeferre', u'Feuilly', u'Prouvaire', u'Grantaire']
7
Information Extraction Architecture
Information comes in many shapes and sizes. One important form is structured data, where there is a regular and predictable organization of entities and relationships Relasilokasi.py: locs = [('Omnicom', 'IN', 'New York'), ('DDB Needham', 'IN', 'New York'), ('Kaplan Thaler Group', 'IN', 'New York'), ('BBDO South', 'IN', 'Atlanta'), ('Georgia-Pacific', 'IN', 'Atlanta')] query = [e1 for (e1, rel, e2) in locs if e2=='Atlanta'] print(query) >>> ['BBDO South', 'Georgia-Pacific'] OrgName LocationName Omnicom New York DDB Needham Kaplan Thaler Group BBDO South Atlanta Georgia-Pacific
8
Simple Information Extraction Systems
Sec. 3.1 Simple Information Extraction Systems Raw text processed using some steps
9
Categorizing and Tagging Words
The process of classifying words into their parts of speech and labeling them accordingly is known as part-of-speech tagging, Here CC, a coordinating conjunction; now and completely are RB(adverbs); for is IN(preposition); something is NN( noun); and different is JJ(adjective). >>> text = word_tokenize("And now for something completely different") >>> nltk.pos_tag(text) [('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ')] Several of the corpora included with NLTK have been tagged for their part-of-speech.
10
Chunking The basic technique we will use for entity detection is chunking, which segments and labels multi-token sequences The smaller boxes show the word-level tokenization and part-of-speech tagging Each of these larger boxes is called a chunk
11
Noun Phrase Chunking we will first define a chunk grammar, consisting of rules that indicate how sentences should be chunked NPchunkparser.py: import nltk, re, pprint sentence = [("the", "DT"), ("little", "JJ"), ("yellow", "JJ"), ("dog", "NN"), ("barked", "VBD"), ("at", "IN"), ("the", "DT"), ("cat", "NN")] grammar = "NP: {<DT>?<JJ>*<NN>}" cp = nltk.RegexpParser(grammar) result = cp.parse(sentence) print(result) result.draw() >>> (S (NP the/DT little/JJ yellow/JJ dog/NN) barked/VBD at/IN (NP the/DT cat/NN))
12
Exploring Text Corpora
We saw how we could interrogate a tagged corpus to extract phrases matching a particular sequence of part-of-speech tags
13
Sec. 3.1 Chinking We can define a chink to be a sequence of tokens that is not included in a chunk. In the following example, barked/VBD at/IN is a chink: [ the/DT little/JJ yellow/JJ dog/NN ] barked/VBD at/IN [ the/DT cat/NN ] Has been used for dictionary lookup in some search engines. Each vocabulary terms(key) is hashed into an integer.
14
Chinking
15
Developing and Evaluating Chunkers
import nltk, re, pprint text = ''' he PRP B-NP accepted VBD B-VP concern NN I-NP . . O ''' nltk.chunk.conllstr2tree(text, chunk_types=['NP']).draw()
16
Conll2000 NP Chunks import nltk, re, pprint from nltk.corpus import conll2000 print(conll2000.chunked_sents('train.txt', chunk_types=['NP'])[99]) >>> (S Over/IN (NP a/DT cup/NN) of/IN (NP coffee/NN) ,/, (NP Mr./NNP Stone/NNP) told/VBD (NP his/PRP$ story/NN) ./.) CoNLL 2000 corpus contains three chunk types: NP chunks, which we have already seen; VP chunks such as has already delivered; and PP chunks such as because of
17
Conll2000 NP Chunks import nltk, re, pprint from nltk.corpus import conll2000 grammar = r"NP: {<[CDJNP].*>+}" cp = nltk.RegexpParser(grammar) test_sents = conll2000.chunked_sents('test.txt', chunk_types=['NP']) print(cp.evaluate(test_sents)) >>> ChunkParse score: IOB Accuracy: 87.7% Precision: 70.6% Recall: 67.8% F-Measure: 69.2% The IOB format (short for Inside, Outside, Beginning) is a common tagging format for tagging tokens in a chunking task in computational linguistics (ex. Named Entity Recognition). The B- prefix before a tag indicates that the tag is the beginning of a chunk, and an I- prefix before a tag indicates that the tag is inside a chunk. An O tag indicates that a token belongs to no chunk.
18
Train it using the CoNLL 2000 corpus
>>> test_sents = conll2000.chunked_sents('test.txt', chunk_types=['NP']) >>> train_sents = conll2000.chunked_sents('train.txt', chunk_types=['NP']) >>> unigram_chunker = UnigramChunker(train_sents) >>> print(unigram_chunker.evaluate(test_sents)) ChunkParse score: IOB Accuracy: 92.9% Precision: 79.9% Recall: 86.8% F-Measure: 83.2%
19
Training classifier base chunker
a. Joey/NN sold/VBD the/DT farmer/NN rice/NN ./. b. Nick/NN broke/VBD my/DT computer/NN monitor/NN ./. These two sentences have the same part-of-speech tags, yet they are chunked differently. In the first sentence, the farmer and rice are separate chunks, while the corresponding material in the second sentence, the computer monitor, is a single chunk
20
Adding features >>> def npchunk_features(sentence, i, history): ... word, pos = sentence[i] ... if i == 0: ... prevword, prevpos = "<START>", "<START>" ... else: ... prevword, prevpos = sentence[i-1] ... return {"pos": pos, "word": word, "prevpos": prevpos} chunker = ConsecutiveNPChunker(train_sents) print(chunker.evaluate(test_sents)) ChunkParse score: IOB Accuracy: 94.5% Precision: 84.2% Recall: 89.4% F-Measure: 86.7%
21
Named Entity Recognition
Named entities are definite noun phrases that refer to specific types of individuals, such as organizations, persons, dates, and so on
22
Named Entity Recognition
NLTK provides a classifier that has already been trained to recognize named entities, accessed with the function nltk.ne_chunk(). If we set the parameter binary=True , then named entities are just tagged as NE; otherwise, the classifier adds category labels such as PERSON, ORGANIZATION, and GPE. >>>print(nltk.ne_chunk(sent)) (S The/DT (GPE U.S./NNP) is/VBZ one/CD ... according/VBG to/TO (PERSON Brooke/NNP T./NNP Mossman/NNP) ...)
23
Relation Extraction Once named entities have been identified in a text, we then want to extract the relations that exist between them Relationentity.py: # Natural Language Toolkit: code_cascaded_chunker import nltk,re, pprint IN = re.compile(r'.*\bin\b(?!\b.+ing)') for doc in nltk.corpus.ieer.parsed_docs('NYT_ '): for rel in nltk.sem.extract_rels('ORG', 'LOC', doc, corpus='ieer', pattern = IN): print(nltk.sem.rtuple(rel)) >>> [ORG: u'WHYY'] u'in' [LOC: u'Philadelphia'] … [ORG: u'BBDO South'] u'in' [LOC: u'Atlanta'] [ORG: u'Georgia-Pacific'] u'in' [LOC: u'Atlanta']
24
XML The Extensible Markup Language (XML) is a markup language much like HTML or SGML. This is recommended by the World Wide Web Consortium and available as an open standard. XML is extremely useful for keeping track of small to medium amounts of data without requiring a SQL-based backbone. XML is the language that is used for communication between Web service components XML Structure
25
XML Structure
26
XML, SOAP and WSDL Standard technologies:
XML. The language (data format) used by the Web service Simple Object Access Protocol (SOAP). SOAP provides a way to communicate between applications running on different operating systems Web Services Description Language (WSDL). An XML file that specifies the location of the service and the operations (or methods) the service exposes.
27
XML for IR The adoption of XML as a standard for the publication and interchange of structured data creates a great opportunity for better information retrieval. XML has the ability to represent the semantics of data in a structured form. It has become crucial to address the question of how we can efficiently query and search large corpora of XML documents. Useful for Enterprise Search , Library Systems and Digital Library
28
XML-Based Enterprise Search
Enterprise Search: the application of IR technology to information finding within organizations Enterprise search may be interpreted as search of digital textual materials owned by an organization, including: search of their external Web site company intranet, and any other electronic text that they hold.
29
APIs in XML (SAX and DOM)
The two most basic and broadly used APIs to XML data are the SAX and DOM interfaces. Simple API for XML (SAX) : It parses the file as it reads it from disk and the entire file is never stored in memory. Document Object Model (DOM) API: This is a World Wide Web Consortium recommendation wherein the entire file is read into memory and stored in a hierarchical (tree-based) form to represent all the features of an XML document.
30
SAX vs DOM SAX obviously cannot process information as fast as DOM can when working with large files. On the other hand, using DOM exclusively can really kill your resources, especially if used on a lot of small files. SAX is read-only, while DOM allows changes to the XML file. Since these two different APIs literally complement each other, there is no reason why you cannot use them both for large projects.
31
Sec. 3.1 XML Structure
32
Similarity and ranking
Sec. 3.1 Similarity and ranking
33
Case study Analytics and Big-Data The Hanging Tree Broken Dreams
We have more than 10,000 books from which we need to search for a book as per the query entered by customer. In addition, we need to create an XML information retrieval system which can call out all the books which resembles the customer query. Here are a few names of books : Analytics and Big-Data The Hanging Tree Broken Dreams Blessed kid Girl with a Dragon Tattoo The query entered by customer is : Book for Analytics newbie.
34
Term Frequency (TF) Matrix
Technique to find out the relevance of a word . Here is a frequency count of a set of words in the 5 books:
35
calculation TF = 1 + log (TF) if TF > 0 0 if TF = 0
Document 1 : = 8.6 Document 2 : + 0 + 2 = 7.3 Document 3 : 2.5 + 3.0 + 0 + 2 = 7.5 Document 4 : 2.6 + 2.3 = 7.9 Document 5 : 2.3 = 7.8
36
calculation Result shows, Document 1 will be more relevant to display for the query, but we still make a concrete conclusion . Since, document 4 and 5 are not far away from Document 1. They might turn out to be relevant too. Document 1 : = 8.6 Document 2 : + 0 + 2 = 7.3 Document 3 : 2.5 + 3.0 + 0 + 2 = 7.5 Document 4 : 2.6 + 2.3 = 7.9 Document 5 : 2.3 = 7.8
37
IDF IDF is another parameter which helps us find out the relevance of words. It is based on the principle that less frequent words are generally more informative. IDF = log (N/DF) where N represents the number of documents and DF represents the number of documents in which we see the occurrence of this word.
38
IDF We now can clearly see that the words like “The” “for” etc. are not really relevant as they occur in almost all the document. Whereas, words like honest, Analytics Big-Data are really niche words which should be kept in the analysis.
39
TF-IDF Matrix As we now know the relevance of words (IDF) and the occurrence of words in the documents (TF), we now can multiply the two. Then, find the subject of the document and thereafter the similarity of query with the document. Now it clearly comes out that document 1 is most relevant to the query “Book for Analytics newbie”.
40
TF-IDF from sklearn.feature_extraction.text import TfidfVectorizer corpus = ["This is very strange", "This is very nice"] vectorizer = TfidfVectorizer(min_df=1) X = vectorizer.fit_transform(corpus) idf = vectorizer._tfidf.idf_ print dict(zip(vectorizer.get_feature_names(), idf))
41
XRANK Architecture The ElemRank Computation module computes the ElemRanks of XML elements . The ElemRanks are then combined with ancestor information to generate an index structure called HDIL (Hybrid Dewey Inverted List). Source: Lin guo et al, XRANK: Ranked Keyword Search over XML Documents, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, pp 16-27, 2003.
42
LXML lxml provides a very simple and powerful API for parsing XML and HTML. It supports one-step parsing as well as step-by-step parsing using an event-driven API (currently only for XML).
43
Example using LXML from lxml import etree from StringIO import StringIO xml = '<a xmlns="test"><b xmlns="test"/></a>' root = etree.fromstring(xml) print etree.tostring(root) >>> <a xmlns="test"><b xmlns="test"/></a>
44
Case study(90 minutes): Use samples of XML Corpus, and:
1. Propose a model for Ranked Keyword Search over XML Documents 2. Display a similarity based on TF-IDF
45
Enterprise Search Architecture
46
Problems in Enterprise Search
The gathering process may take a long time and generate telecommunications charges Filtering of large collections of binary format documents may be very time consuming Formats such as MSWord, PDF and JPEG are capable of storing metadata such as title, author, subject, and date in practice, however, these metadata are usually missing
47
Environment in Library
48
OPAC
49
Exercise Read 1 document and display the relation using Relation Extraction. Example PERSON in ORGANIZATION
50
References Ricardo Baeza-Yates and Berthier ribeiro-Neto (2011), Modern Information Retrieval, 2nd edition, ACM Books Press. ISBN: Stefan Buttcher, Charles L.A Clarke and Gordon V. Cormack (2010), Information Retrieval- Implementing and Evaluating Search Engines, MIT Press, ISBN: NLTK Book, Python.org Raymond J. Mooney, Mining Knowledge from Text Using Information Extraction, SIGKDD Explorations., vol 7(1). Conference on Computational Natural Language Learning (CoNLL-2000)
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.