ALTA Workshop’04, Macquarie University, Sydney 8 December 2004 Luiz Augusto Sangoi Pizzato Using a Trie-based Structure for Question Analysis

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (2/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Question analysis Trie structure Question trie Building and retrieving using the trie Evaluation of the Technique Further work “Using a Trie-based Structure for Question Analysis” Outline

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (3/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Our question analyser tries to answer two meta-questions: What is the kind of answer I have to provide? Define the expected answer type (EAT). What is the subject of the question? Define the question focus. Question on question

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (4/21) In: ALTA Workshop Macquarie University, Sydney. 8 December EAT Handcrafted rules Normally by the use of RE WordNet top concepts (Moldovan et al., 2003) High quality results Support Vector Machines (SVM) (Zhang and Lee, 2003) Good results using a large training set Focus Discard question’ stopwords. Some approaches

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (5/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Trie structure a|b|c|d|e|f|...|z a|b|c|d|...|r|...|z car a|b|c|d|e|f|...|z a|b|c|d|...|r|...|z a|b|c|d|e|f|...|z zebra

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (6/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Patterns QuestionPatternEAT Where is Chile?^ Where is !LOC $LOC Who is the dean of ICS?^ Who is the !POS of !ORG $NAME Who is J. Smith?^ Who is !NAME $DESC Who is J. Smith of ICS?^ Who is !NAME of !ORG $DESC How far is Athens?^ How far is !LOC $NO How tall is Sting?^ How tall is !NAME $NO

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (7/21) In: ALTA Workshop Macquarie University, Sydney. 8 December where 6 who 18 how 7 is 13 !NAME 9 !POS 10 of 11 !ORG 12 $ (eoq) 14 $ (eoq) 8 the 15 of 16 !ORG 17 $ (eoq) 3 is 4 !LOC 5 $ (eoq) 19 far 20 is 21 !LOC 22 $ (eoq) 23 tall 24 is 25 !NAME 26 $ (eoq) ^ (boq) Question Trie

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (8/21) In: ALTA Workshop Macquarie University, Sydney. 8 December NodesInformation (EAT, Frequency) 1(LOC,1),(NAME,1),(DESC,2),(NUMBER,2) 2-5(LOC,1) 6-7(NAME,1),(DESC,2) 8-12(NAME,1) 13(DESC,2) 14-17(DESC,1) 18(NUMBER,2) 19-26(NUMBER,1) 1 2 where 6 who 18 how 7 is 13 !NAME 9 !POS 10 of 11 !ORG 12 $ (eoq) 14 $ (eoq) 8 the 15 of 16 !ORG 17 $ (eoq) 3 is 4 !LOC 5 $ (eoq) 19 far 20 is 21 !LOC 22 $ (eoq) 23 tall 24 is 25 !NAME 26 $ (eoq) ^ (boq) Question Trie

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (9/21) In: ALTA Workshop Macquarie University, Sydney. 8 December who 7 is 13 !NAME 14 $ (eoq) 15 of 16 !ORG 17 $ (eoq) ^ (boq) $^whoisJohnSmithofMacquarieUniversity ? ? $^whoisMadonna ? Look-ahead process

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (10/21) In: ALTA Workshop Macquarie University, Sydney. 8 December JustAsk logs; 4.8% NL questions of were NL questions unique NL questions 23% with some language problems: Why this search not word? Unusual language: Do u offer any scholarships 4 physiotherapy? Speculative questions: Will I get a job in Australia after finishing my MBA? MQ Questions

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (11/21) In: ALTA Workshop Macquarie University, Sydney. 8 December JustAsk questions were randomly selected and semi- automatically tagged according to a XML like structure Who is Luiz Pizzato ? Total number of questions: – Who 212 – What 208 – Where 203 – How 529 – Other types: Am I, Are there, Can I, Do you, Is there, I want, I need, Which, Does, Tell me, Why, Have you, Could you, May I, Will I, Was I, Would you, Whom Training Set

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (12/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Evaluation - EAT

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (13/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Evaluation – Focus

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (14/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Question Trie without Entities 1 2 where 6 who 19 how 7 is 13 J. 9 dean 10 of 11 ICS 12 $ (eoq) 15 $ (eoq) 8 the 16 of 17 ICS 18 $ (eoq) 3 is 4 Chile 5 $ (eoq) 20 far 21 is 22 Athens 23 $ (eoq) 24 tall 25 is 26 Sting 27 $ (eoq) ^ (boq) 14 Smith

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (15/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Evaluation – TREC-2003

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (16/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Comparison with SVM (Zhang and Lee, 2003)

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (17/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Concluding remarks The developed technique offers reasonable results using no linguistic resources. Future developments Define guidelines for the EAT markup and review the markup of the MQ questions Adding POS and semantic information from WordNet may replace entity markup

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (18/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Combine lexical and POS information Who is John Smith? is VBZ EAT freq NAME1 DESC1 Who WP EAT freq NAME1 DESC1^ EAT freq NAME1 DESC1 $ EAT freq NAME1 John NNP EAT freq NAME1 Smith NNP EAT freq NAME1 John Smith NNP EAT freq NAME 1 Further Work

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (19/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Dell Zhang and Wee Sun Lee Question classification using support vector machines. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR-03), pages 26–32. ACM Press. Dan Moldovan, Marius Paşca, Sanda Harabagiu, and Mihai Surdeanu Performance issues and error analysis in an open-domain question answering system. ACM Trans. Inf. Syst., 21(2):133–154. References

Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (20/21) In: ALTA Workshop Macquarie University, Sydney. 8 December Acknowledgments My supervisors Dr. Diego Mollá-Aliod Dr. Rolf Schwitter Dr. Cecile Paris

ALTA Workshop’04, Macquarie University, Sydney 8 December 2004 Luiz Augusto Sangoi Pizzato Using a Trie-based Structure for Question Analysis