Is Question Answering an Acquired Skill? Soumen Chakrabarti G. Ramakrishnan D. Paranjpe P. Bhattacharyya IIT Bombay

QAChakrabarti3 Web search and QA Information need – words relating things + thing aliases = telegraphic Web queries Cheapest laptop with wireless best price laptop Why is the sky blue? sky blue reason When was the Space Needle built? Space Needle history Entity and relation extraction technology better than ever (SemTag, KnowItAll) Ontology extension (e.g., is a kind of) List extraction (e.g., is an instance of) Slot-filling (author X wrote book Y)

QAChakrabarti4 Factoid QA Specialize given domain to a token related to ground constants in the query What animal is Winnie the Pooh? hyponym(animal) NEAR Winnie the Pooh When was television invented? instance-of(time) NEAR television NEAR synonym(invented) Three kinds of useful question tokens Appear unchanged in passage (selector) Specialize to answer tokens (atype) Improve belief in answer via synonymy etc.

QAChakrabarti5 A new relational view of QA Entity class or atype may be expressed by A finite IS-A hierarchy (e.g. WordNet, TAP) A surface pattern matching infinitely many strings (e.g. digit+, Xx+, preceded by a preposition) Match selectors, specialize atype to answer tokens QuestionAtype clues Selectors Answer passage Question words Landing zone Direct syntactic match Entity class IS-A Limit search to certain rows Locate which column to read Landing zone Attribute or column name

QAChakrabarti6 Benefits of the relational view Scaling up by dumbing down Next stop after vector-space Far short of real knowledge representation and inference Barely getting practical at (near) Web scale Can set up as a learning problem: train with questions and answers embedded in passage context Transparent, self-tuning, easy to deploy Feature extractors used in entity taggers Relational/graphical learning on features

QAChakrabarti7 Subproblems Identify atype clues Easy: who, when, where, how many, how tall… Harder: What…, which…, name… Map atype clues to likely entity classes Data- and task-driven question classification Train quickly on new corpus and QA samples Identify selectors for keyword query Based on question context and global stats Get candidate passages from IR system Re-rank candidate passages

QAChakrabarti8 Mapping self-evident atypes Who person, when time, where place Not always trivial: how_many vs. when Question classification + handcrafted map Needs task knowledge and skilled effort Laborious to move to new corpus, language… Task-driven information extraction Enough info in training QA pairs to learn map Map clue to a generalization of the answer Surface patterns: hasDigit, [in] DDDD, NNP, CD WordNet-based: region#n#3, quantity#n#1

QAChakrabarti9 Mapping examples howwho fastmanyfarrich wrotefirst How fast can a cheetah run? A cheetah can chase its prey at up to 90 km/h How fast does light travel? Nothing moves faster than 186,000 miles per hour, the speed of light rate#n#2 abstraction#n#6 NNS rate#n#2 magnitude_relation#n#1 mile#n#3 linear_unit#n#1 measure#n#3 definite_quantity#n#1 paper_money#n#1 currency#n#1 writer, composer, artist, musician NNP, person explorer WordNet

QAChakrabarti10 What…, which…, name… atype clues Assumption: Question sentence has a wh- word and a main/auxiliary verb Observation: Atype clues are embedded in a noun phrase (NP) adjoining the main or auxiliary verb Heuristic: Atype clue = head of this NP Use a shallow parser and apply rule Head can have attributes Which (American (general)) is buried in Salzburg? Name (Saturns (largest (moon)))

QAChakrabarti11 Atype clue extraction stats Simple heuristic surprisingly effective If successful, extracted atype is mapped to WordNet synset (moon celestial body etc.) If no atype of this form available, try the self- evident atypes (who, when, where, how_X etc.)

QAChakrabarti12 Learning selectors Which question words are likely to appear (almost) unchanged in an answer passage? Constants in select-clauses of SQL queries Guides backoff policy for keyword query Local and global features POS of word, POS of adjacent words, case info, proximity to wh-word Suppose word is associated with synset set S NumSense: size of S (how polysemous is the word?) NumLemma: average #lemmas describing s S

QAChakrabarti13 Selector results Decision trees better than logistic regression F1=81% as against LR F1=75% Intuitive decision branches But logistic regression gives scores for query backoff Global features (IDF, NumSense, NumLemma) essential for accuracy Best F1 accuracy with local features alone: 7173% With local and global features: 81%

QAChakrabarti14 Putting together a QA system QA System Wordnet POS Tagger Training Corpus Shallow parser Learning tools N-E Tagger

QAChakrabarti15 Question Passage Index Corpus Sentence splitter Passage indexer Candidate passage Keyword query Keyword query generator Shallow Parser Noun and verb markers Atype Extractor Atype clues Learning to rerank passages Sample features: Do selectors match? How many? Is some non-selector passage token a specialization of the questions atype clue? Min, avg linear token distance between candidate token and matched selectors Learning to rerank passages Sample features: Do selectors match? How many? Is some non-selector passage token a specialization of the questions atype clue? Min, avg linear token distance between candidate token and matched selectors Logistic Regression Reranked passages Putting together a QA system Tokenizer POS Tagger Tagged question Tokenizer POS Tagger Entity Extractor Tagged passage Selector Learner Is QA pair?

QAChakrabarti16 Learning to re-rank passages Remove passage tokens matching selectors User already knows these are in passage Find passage token/s specializing atype For each candidate token collect Atype of question, original rank of passage Min, avg linear distances to matched selectors POS and entity tag of token if available Ushuaia, a port of about 30,000 dwellers set between the Beagle Channel and … How many inhabitants live in the town of Ushuaia selector match Surface pattern hasDigits WordNet match 5 tokens apart1

QAChakrabarti17 Effect of re-ranking results Categorical and numeric attributes Logistic regression Good precision, poor recall Use logit score to re-rank passages Rank of first correct passage shifts substantially Log scale

QAChakrabarti18 Mean reciprocal rank studies n q = smallest rank among answer passages Re-ranking reduces n q drastically MRR = (1/|Q |) q Q (1/n q ) Substantial gain in MRR TREC 2000 top MRRs:

QAChakrabarti19 Generalization across corpora Across-year numbers close to train/test split on a single year Features and model seem to capture corpus- independent linguistic Q+A artifacts

QAChakrabarti20 Re-ranking benefits by question type All question types benefit from re- ranking Benefits differ by question type Large benefits for what and which questions, thanks to WordNet Without WordNet customization

QAChakrabarti21 Conclusion A clean-room view of QA as feature extraction plus learning Recover structure info from question Learn correlations between question structure and passage features Competitive accuracy with negligible domain expertise or manual intervention Ongoing work Use redundancy available from the Web Model how selector and atype are related Treat all question types uniformly