Question Answering Passage Retrieval Using Dependency Relations (SIGIR 2005) (National University of Singapore) Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan, Tat-Seng Chua

2 Abstract Problems: Relationships among Q terms are not considered. Strict matching is used, but semantically equivalent relationships are phrased differently. Solution: fuzzy relation matching based on statistical models. Mutual information Expectation maximization Results: Significantly outperforms by up to 78% in MRR. Brings a 50% improvement enhanced by query expansion.

3 Introduction (1/3) QA steps: Locating the relevant documents. Retrieving passages that may contain the A. Pinpointing the exact A from candidate passages. Passage retrieval greatly affects the performance of a QA system. Too many irrelevant passages  fail to pinpoint the correct A due to too much noise. Passages provide sufficient context to understand the A.

4 Introduction (2/3) Importance of relationships: Neglecting crucial relations b/w words is a major source of false positives for current lexical matching based retrieval techniques. Many irrelevant passages share the same question terms w/ correct ones, but different relationships. Example

5 Introduction (3/3) Fuzzy relation matching method: Examines grammatical dependency relations b/w Q terms. Minipar: dependency parser. Statistical technique: measuring the degree of match of candidate sentences and Q. Fuzzy matching: b/c the same relationship is often phrased differently of Q&A. Training: MI & EM.

6 Related Work Passage retrieval: Count the # of matched Q terms. Consider the distance b/w Q terms in passages.  Dependency relations exist b/w words. Approximate such relations statistically: Using bigrams.  Only b/w adjacent words. Select answers: Strict matching.  Words often differ b/w Q and A sentences.

7 Fuzzy Relation Matching for Passage Retrieval Extracting and pairing relation paths. Present how relation paths are extracted and paired from parse trees. Measuring path matching score. A variation of IBM translation model. Model training: learn the mapping scores b/w relations in Q and potential A sentences. MI: capture the bipartite co-occurrences of 2 relations. EM: iterative training process.

8 1. Extracting and Pairing Relation Paths Dependency trees: generated by Minipar. Dependency trees Node: a word / a chunked phrase. Link: relation Ignore the directions of relations. (roles can change in Q & A) Paths: 2 constraints Paths The path length cannot exceed a pre-defined threshold (7). Minipar only resolves nearby dependencies reliably. Ignore relation paths b/w 2 words if they belong to the same chunk. “ 28 percent ”, “ New York ”

9 2. Measuring Path Matching Score Measure the matching score of the paths extracted from the sentence according to those from the question. Matching their nodes at both ends. Extending IBM statistical translation model 1. Matching score of a relation path from a candidate sentence = the probability of translating to it from it ’ s corresponding path in the Q. Sum up the matching scores of each path from the sentence which has a corresponding path in the Q.

10 3. Model Training Relation translation probability 2 methods (MI & EM). Mutual information Measured by their bipartite co-occurrences. Expectation maximization: Employ GIZA, a publicly available statistical translation package. Perform an iterative training process using EM to learn pair-wise translation probabilities.

11 Evaluations (1/5) Experiment setup: Training data: 10,255 factoid QA pairs from TREC- 8 and 9 QA tasks. Test data: TREC-12. (324 QA pairs) 4 systems for comparison: MITRE (baseline) Strict Matching of Relations SiteQ NUS 3 performance metrics: MRR. % of Q that have NO correct A. Precision at the top one passage.

12 Evaluations (2/5) Correct: Must be matched by the exact answer pattern. Must have a cosine similarity equal to or above.75 w/ any standard answer passage. Evaluation observations: Overall performance. Performance variation to Q length. Error Analysis for Relation Matching. Performance w/ Query Expansion.

13 Evaluations (3/5) Overall performance of relation matching technique compared to other passage retrieval systems. Overall performance Relation matching HELPS. Fuzzy outperforms strict matching. MI ≈ EM Performance variation to Q length: longer Qs are likely to benefit more from relation matching than shorter Q. Performance variation to Q length

14 Evaluations (4/5) Error Analysis for Relation Matching: Mismatch of Q terms: paths are incorrectly paired due to the mismatch of Q terms. Q: “ In which city is the River Seine? ” A: “ Paris. ” Paraphrasing b/w Q and A sentences: some correct sentences are paraphrases of the given Q. Q: “ What company manufactures X? ” A: “… C, the manufacturer of X …” Paraphrase: “ C is the manufacturer of X  C manufactures X ”

15 Evaluations (5/5) Performance w/ Query Expansion (QE): Short Q and paraphrases are obstacles in enhancing performance using relation matching. Submit Q to Google and select expansion terms based on their co-occurrences w/ Q terms in result snippets. External expansion terms do not have relation paths w/ the original Q terms in the Q.

16 Future Work Future work: Larger amount of training data. (scalability of MI & EM) Larger amount of test data. (effect of # of relation paths on matching) Question analysis and typing. Seamless integration of query expansion w/ relation matching. (take advantage of expanded terms in relation matching)

17 Conclusion Present a novel fuzzy relation matching technique (based on the degree of match b/w relation paths in candidate sentences and Q) for factoid QA passage retrieval. Produce significant improvements in retrieval performance in current systems: a vast 50~138% improvement in MRR, and over 95% in precision at top one passage.

18 ThanQ

19 Lexical Matching

20 Dependency Tree

21 Relation Paths

22 Note: Measuring Path Matching Score (1/2) Translation probability Prob(P S | P Q ): the sum over all possible alignments. P Q : corresponding paths from question Q. (length: m) P S : corresponding paths from sentence S. (length: n) Rel i (S) : ith relation in path P S. Relα i (Q) : corresponding relation in path P Q. P t (Rel i (S) |Relα i (Q) ): relation mapping scores.

23 Note: Measuring Path Matching Score (2/2) Consider only the most probable alignment: A i : the most probable alignment. Use only n for normalization. m is constant for the same Q.

24 Note: Model Training - MI P t (Rel i (S) |Relα i (Q) ): 2 methods (MI & EM) Mutual information: measured by their bipartite co- occurrences. : 1 when Rel i (S) & Rel j (Q) appear together in a training path pair, and 0 otherwise. γ: the inverse proportion of the sum of the lengths of the 2 paths. | Rel (Q) |: the # of paths extracted from all Q in which relation Rel occurs. | Rel (S) |: the # of paths extracted from all A sentences in which relation Rel occurs.

25 Overall Performance

26 Performance Variation to Q Length (Along w/ 95% error bars)

27 Query Expansion