1. 18-12-07 | 1/21 › Off-line answer extraction / Jori Mur Question Answering

2. 18-12-07 | 2/21 Overview ›Using patterns to extract answers off-line ›High precision, low recall Simple patterns not robust enough Answer not in one sentence No binary relation in question Hard to come up with new patterns

3. 18-12-07 | 3/21 Joost

4. 18-12-07 | 4/21 Off-line answer extraction using patterns ›/[Noun] [Name]/... premier Balkenende... ›/[Name], [Noun]/... Balkenende, premier... ›/[Name] [Verb] as [Noun]/... Balkende heeft als premier... ›/[Name] [be] [Det] [Noun]/... Balkenende is de premier... ›...

5. 18-12-07 | 5/21 High precision, low recall ›Tjong Kim Sang, Bouma en de Rijke. Developing Offline Strategies for Answering Medical Questions, 2005 ›Bernardi, Jijkoun, Mishne en de Rijke. Selectively Using Linguistic Resources Throughout the Question Answering Pipeline, 2003

6. 18-12-07 | 6/21 Investigating experiment ›Joost ›99 function questions from CLEF2003 and CLEF2004 Who was the author of Die Gotterdämmerung? Of which company is Christian Blanc president? ›15 patterns /[ADJ][NOUN][NAME]/ Franse president Sarkozy /[NAME], [NOUN] van [COUNTRY]/ Sarkozy, president van Frankrijk

7. 18-12-07 | 7/21 Results ›Extraction ›QA

8. 18-12-07 | 8/21 Error analysis: 46 No answer found ›In which company is Christian Blanc president? ›Christian Blanc, president of the rather destitute French aviation company Air France ›Who is the daughter of Deng Xiaopeng? ›Will Deng Xiaoping make it until the Chinese New Year? (...) when Deng's youngest daughter, Deng Rong, recently said (...) ›Who is the head of state of Australia? ›The prime-minister of Australia explained his plan to transfer the role of the head of state from the British queen Elizabeth to (...)

9. 18-12-07 | 9/21 Error analysis: 18 Incorrect ›Who is the German minister of Economy? ›Who was the president of the US during the second World War?

10. 18-12-07 | 10/21 4 Challenges ›Simple patterns are not robust enough ›Not easy to think of new patterns ›Answer not in one sentence ›Not always binary relations

11. 18-12-07 | 11/21 1. Simple patterns are not robust enough... George Bush, yesterday he became 61, said today as 43rd president of the US...... George Bush said as president of the US... PredmSubjObj Mod Subj Predm Obj Mod ›

12. 18-12-07 | 12/21 2. Answer not in one sentence Balkenende was happy with the result, allthough he did not win the elections.

13. 18-12-07 | 13/21 2. Evaluation and Results Coreference MUC score : |S|-|p(S)| |S| -1 Recall: 45.6% Precision: 67.9% Extraction Without coreference With coreference 127.711 facts 146.210 facts QA Without coreferenceWith coreference 153/233 (65.7%) 159/233 (68.2%) Question 20050107: Wie was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde? ‘Who was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope?’. The answer in the AD of September 19th, 1995: Bowersox was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde. ‘Bowersox was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope’.

14. 18-12-07 | 14/21 3. Not always binary relations ›When was Jimi Hendrix born? Jimi Hendrix – 1942 ›Who is the president of Amerika? Bush – president – Amerika ›Who was the president of Amerika in 1944? Roosevelt – president – Amerika – 1944 ›Who succeeded president Roosevelt as president of Amerika in the second world war? Truman – president – Amerika – WO II Truman – successor – Roosevelt

15. 18-12-07 | 15/21 3. Dependency relation overlap Q: Who succeeded president Roosevelt as president of Amerika in the second world war? A1: George Bush, president of Amerika said that... A2: At the end of the second world war Harry S. Truman succeeded president Roosevelt as president of Amerika. A1: A2:

16. 18-12-07 | 16/21 4. Hard to transfer to new question type ›Learn patterns in bootstrapping way 1. Start with 10 seed-pairs 2. Find dependency path between two terms 3. Replace terms by variables to create pattern 4. Use patterns to find new facts 5. Use new facts to find new patterns Find balance precision/recall 3a. Filter patterns

17. 18-12-07 | 17/21 4. Calculate precision patterns ›Replace only the answer term with a variable. ›P = Ca/Co ›Co : the total number of times the pattern matched a phrase in the corpus ›Ca : the total number of times the pattern matched a phrase in the corpus containing the correct answer term

18. 18-12-07 | 18/21 4. Experiment and Results learning ›Stop at precision of table < 0.5 or more than 100 patterns ›24 capital question from Clef, 3 turned out NIL

19. 18-12-07 | 19/21 4. Nice examples of found patterns ›... is na H de grootste stad van L... Petersburg is na Moskou de grootste stad van Rusland ›... de L-adj tenniskampioenschappen in H... de open Franse tenniskampioenschappen in Parijs ›... de L-adj president opent in H... De Tjechische president Havel opent op 14 juni in Praag ›... de L-adj ministers komen bijeen in H... Volgende maand komen de Europese minister van industrie opnieuw bijeen in Brussel

20. 18-12-07 | 20/21 Summary ›Qatar – off-line answer extraction module in Joost High precision, low recall Dependency patterns Coreference resolution Score for dependency overlap Learning patterns

21. 18-12-07 | 21/21 Questions?