1. ALTA Workshop’04, Macquarie University, Sydney 8 December 2004 Luiz Augusto Sangoi Pizzato pizzato@ics.mq.edu.au http://www.ics.mq.edu.au/~pizzato Using a Trie-based Structure for Question Analysis

2. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (2/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Question analysis Trie structure Question trie Building and retrieving using the trie Evaluation of the Technique Further work “Using a Trie-based Structure for Question Analysis” Outline

3. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (3/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Our question analyser tries to answer two meta-questions: What is the kind of answer I have to provide? Define the expected answer type (EAT). What is the subject of the question? Define the question focus. Question on question

4. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (4/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. EAT Handcrafted rules Normally by the use of RE WordNet top concepts (Moldovan et al., 2003) High quality results Support Vector Machines (SVM) (Zhang and Lee, 2003) Good results using a large training set Focus Discard question’ stopwords. Some approaches

5. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (5/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Trie structure a|b|c|d|e|f|...|z a|b|c|d|...|r|...|z car a|b|c|d|e|f|...|z a|b|c|d|...|r|...|z a|b|c|d|e|f|...|z zebra

6. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (6/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Patterns QuestionPatternEAT Where is Chile?^ Where is !LOC $LOC Who is the dean of ICS?^ Who is the !POS of !ORG $NAME Who is J. Smith?^ Who is !NAME $DESC Who is J. Smith of ICS?^ Who is !NAME of !ORG $DESC How far is Athens?^ How far is !LOC $NO How tall is Sting?^ How tall is !NAME $NO

7. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (7/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. 1 2 where 6 who 18 how 7 is 13 !NAME 9 !POS 10 of 11 !ORG 12 $ (eoq) 14 $ (eoq) 8 the 15 of 16 !ORG 17 $ (eoq) 3 is 4 !LOC 5 $ (eoq) 19 far 20 is 21 !LOC 22 $ (eoq) 23 tall 24 is 25 !NAME 26 $ (eoq) ^ (boq) Question Trie

8. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (8/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. NodesInformation (EAT, Frequency) 1(LOC,1),(NAME,1),(DESC,2),(NUMBER,2) 2-5(LOC,1) 6-7(NAME,1),(DESC,2) 8-12(NAME,1) 13(DESC,2) 14-17(DESC,1) 18(NUMBER,2) 19-26(NUMBER,1) 1 2 where 6 who 18 how 7 is 13 !NAME 9 !POS 10 of 11 !ORG 12 $ (eoq) 14 $ (eoq) 8 the 15 of 16 !ORG 17 $ (eoq) 3 is 4 !LOC 5 $ (eoq) 19 far 20 is 21 !LOC 22 $ (eoq) 23 tall 24 is 25 !NAME 26 $ (eoq) ^ (boq) Question Trie

9. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (9/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. 16 who 7 is 13 !NAME 14 $ (eoq) 15 of 16 !ORG 17 $ (eoq) ^ (boq) $^whoisJohnSmithofMacquarieUniversity ? ? $^whoisMadonna ? Look-ahead process

10. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (10/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. JustAsk logs; 4.8% NL questions 60.732 of 1.275.116 were NL questions 47.844 unique NL questions 23% with some language problems: Why this search not word? Unusual language: Do u offer any scholarships 4 physiotherapy? Speculative questions: Will I get a job in Australia after finishing my MBA? MQ Questions

11. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (11/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. JustAsk questions were randomly selected and semi- automatically tagged according to a XML like structure Who is Luiz Pizzato ? Total number of questions: 1385 233 – Who 212 – What 208 – Where 203 – How 529 – Other types: Am I, Are there, Can I, Do you, Is there, I want, I need, Which, Does, Tell me, Why, Have you, Could you, May I, Will I, Was I, Would you, Whom Training Set

12. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (12/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Evaluation - EAT

13. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (13/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Evaluation – Focus

14. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (14/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Question Trie without Entities 1 2 where 6 who 19 how 7 is 13 J. 9 dean 10 of 11 ICS 12 $ (eoq) 15 $ (eoq) 8 the 16 of 17 ICS 18 $ (eoq) 3 is 4 Chile 5 $ (eoq) 20 far 21 is 22 Athens 23 $ (eoq) 24 tall 25 is 26 Sting 27 $ (eoq) ^ (boq) 14 Smith

15. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (15/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Evaluation – TREC-2003

16. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (16/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Comparison with SVM (Zhang and Lee, 2003)

17. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (17/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Concluding remarks The developed technique offers reasonable results using no linguistic resources. Future developments Define guidelines for the EAT markup and review the markup of the MQ questions Adding POS and semantic information from WordNet may replace entity markup

18. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (18/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Combine lexical and POS information Who is John Smith? is VBZ EAT freq NAME1 DESC1 Who WP EAT freq NAME1 DESC1^ EAT freq NAME1 DESC1 $ EAT freq NAME1 John NNP EAT freq NAME1 Smith NNP EAT freq NAME1 John Smith NNP EAT freq NAME 1 Further Work

19. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (19/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Dell Zhang and Wee Sun Lee. 2003. Question classification using support vector machines. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR-03), pages 26–32. ACM Press. Dan Moldovan, Marius Paşca, Sanda Harabagiu, and Mihai Surdeanu. 2003. Performance issues and error analysis in an open-domain question answering system. ACM Trans. Inf. Syst., 21(2):133–154. References

20. Pizzato, Luiz Augusto Sangoi. Using a Trie-based Structure for Question Analysis. (20/21) In: ALTA Workshop 2004. Macquarie University, Sydney. 8 December 2004. Acknowledgments My supervisors Dr. Diego Mollá-Aliod Dr. Rolf Schwitter Dr. Cecile Paris

21. ALTA Workshop’04, Macquarie University, Sydney 8 December 2004 Luiz Augusto Sangoi Pizzato pizzato@ics.mq.edu.au http://www.ics.mq.edu.au/~pizzato Using a Trie-based Structure for Question Analysis