ANLE1 CC 437: Advanced Natural Language Engineering ASSIGNMENT 2: Implementing a query expansion component for a Web Search Engine

ANLE 2 Goal of this class We’ll go in more detail over the assignment SW that may be used.

ANLE 3 The system you have to implement Input: A string of words (possibly a complete sentence) LIST THE ESTATE AGENTS IN STRATFORD, LONDON. I AM LOOKING FOR A CAR MECHANIC IN WIVENHOE Minimum Output: a query for a Web search engine (“ESTATE AGENT” OR PROPERTY OR “REAL ESTATE”) AND STRATFORD AND LONDON Possible extension (10%): Actually access search engine E.g., GOOGLE: %22+OR+%22real+estate%22+OR+property

ANLE 4 Reminder: the basic pipeline in IE systems PREPROCESSING LEXICAL PROCESSING SYNTACTIC PROCESSING SEMANTIC PROCESSING DISCOURSE PROCESSING

ANLE 5 The pipeline for a query expansion system PREPROCESSING LEXICAL PROCESSING SYNTACTIC PROCESSING SEMANTIC PROCESSING WEB ACCESS List the estate agents in Stratford, London. TOKENIZATION POS TAGGING TERM IDENTIFICATION STOP WORDS SYNONYMS

ANLE 6 Processing Steps, II Preprocessing: Possibly: eliminate stop words LIST THE ESTATE AGENTS IN STRATFORD LONDON Possibly: XML markup

ANLE 7 Preprocessing, I: tokenizing List the estate agents in Stratford, London PARAGRAPH MARKUP; TOKENIZER List the estate agents in Stratford, London

ANLE 8 Processing Steps, II LEXICAL PROCESSING: POS TAGGING THE -> THE/DT; ESTATE -> ESTATE/NN STEMMING / LEMMATIZATION AGENTS -> AGENT (or even: AGENT + N +PL)

ANLE 9 Lexical Processing, I: POS tagging List the estate agents in Stratford, London

ANLE 10 Lexical Processing, II: lemmatizing / stemming List the estate agent in Stratford, London

ANLE 11 Processing Steps, II SYNTACTIC PROCESSING: Identify terms: “ESTATE AGENT” Remove stopwords (e.g., words tagged as DT, IN, VB, … )

ANLE 12 Practical (partial) parsing: identifying search terms, filtering estate agent Stratford, London

ANLE 13 Processing Steps, II SEMANTIC PROCESSING: “ESTATE AGENT” OR PROPERTY QUERY FORMATION: Abstract query Concrete query

ANLE 14 Semantic processing: finding synonyms, (or better keywords); interpreting stop words. estate agent real estate Stratford, London

ANLE 15 Available tools: LINUX: Overall system control: Shell scripts, Perl, Java Tokenizing: Java / Perl + Regular Expressions POS: Brill tagger, QTAG Lexical Expansion: WordNet (Java interface, command line) WINDOWS: Overall system control: Java, Batch files, Perl Tokenizing: Java / Perl + Regular expressions Tokenizing, POS tagging: Connexor (Tokenizer, POS + Lemmatizer) POS: QTAG WordNet: Use Java interface

ANLE 16 Marking Scheme Engineering a complete system that takes input, produces output, and calls the appropriate modules 20% Pre-processing (tokenizing, normalization)15% Part-of-speech tagging15% Removing stopwords15% Lexical expansion using WordNet15% Report10% Calling a search engine10% Total100%

ANLE 17 Optionals Write a simple Web page interface to your search engine Write your own lexical resource (see following classes)

ANLE 18 Deadline Friday, December 16 th, 12:00