1. JAVELIN Project Briefing 1 AQUAINT Year I Mid-Year Review Language Technologies Institute Carnegie Mellon University Status Update for Mid-Year Program Review May 15, 2002

2. JAVELIN Project Briefing 2 AQUAINT Year I Mid-Year Review Eric Nyberg Jamie CallanJaime Carbonell Bob Frederking Alon Lavie Teruko MitamuraDave Svoboda Jeongwoo Ko Michael DugganKrzysztof CzubaVasco PedroYifen Huang Laurie Hiyakumoto Lucian Lita Current Team

3. JAVELIN Project Briefing 3 AQUAINT Year I Mid-Year Review Research Objectives QA as Planning –Create a general QA planning system –How should a QA system represent its chain of reasoning? QA and Auditability –How can we improve a QA system’s ability to justify its steps? –How can we make QA systems open to machine learning?

4. JAVELIN Project Briefing 4 AQUAINT Year I Mid-Year Review Research Objectives [2] Utility-Based Information Fusion –Perceived utility is a function of many different factors –Create and tune utility metrics, e.g.: U = Argmax k [F (Rel(I,Q,T), Nov(I,T,A), Ver(S,Sup(I,S)), Div(S), Cmp(I,A)), Cst(I,A)] I: Info item, Q: Question, S: Source, T: Task context, A: Analyst - relevance - novelty - veracity, support - diversity - comprehensibility - cost

5. JAVELIN Project Briefing 5 AQUAINT Year I Mid-Year Review Project Status Summary Started in November 2001 Fully staffed in December 2001 Basic end-to-end architecture operational w/limited coverage Working on question types for TREC evaluation Working to integrate advance JAVELIN capabilities (e.g., Planner)

6. JAVELIN Project Briefing 6 AQUAINT Year I Mid-Year Review Research Plan & Progress Develop end-to-end system –Architecture, Planner, Repository –Individual QA Modules First version being tested now Evaluation: –English queries –English, Chinese and Japanese documents Starting with English only

7. JAVELIN Project Briefing 7 AQUAINT Year I Mid-Year Review QA Evaluation in the Large Information-Based Evaluation current focus, for TREC QA track Utility-Based Evaluation when planner is integrated Architectural Evaluation future task: specify and analyze the properties of the design JAVELIN team participated in LREC ’02 workshop on QA systems evaluation

8. JAVELIN Project Briefing 8 AQUAINT Year I Mid-Year Review JAVELIN Basic Architecture All objects created or retrieved are stored centrally for reuse Details of module implementation are hidden Planner is independent from the particular QA modules being used First End-to-End System Completed Next Step: Integrate Planner

9. JAVELIN Project Briefing 9 AQUAINT Year I Mid-Year Review JAVELIN Data Flow Merged

10. JAVELIN Project Briefing 10 AQUAINT Year I Mid-Year Review Revised Architecture Data Repository JAVELIN GUI Question Analyst Answer Generator Retrieval Strategist Execution Manager... search engines & document collections process history and results operator (action) models Request Filler Planner Domain Model

11. JAVELIN Project Briefing 11 AQUAINT Year I Mid-Year Review Module Integration Via XML DTDs for each object type Modules use simple XML object-passing protocol built on TCP/IP Execution Manager takes care of checking objects in/out of Repository

12. JAVELIN Project Briefing 12 AQUAINT Year I Mid-Year Review End-to-End System User Interface Repository Execution Manager Question Analyzer Retrieval Strategist Request Filler Answer Generator Planner Java client Microsoft SQL Server Java server/IIM Module server/KANTOO Module server/Inquery Module server

13. JAVELIN Project Briefing 13 AQUAINT Year I Mid-Year Review Evaluation: Current Execution Manager can run in “lights out” batch mode Nightly tests on different test suites (starting with TREC QA) Results include scores and logs for debugging Working up to full TREC QA evaluation for TREC ‘02

14. JAVELIN Project Briefing 14 AQUAINT Year I Mid-Year Review Sample Test Results Page

15. JAVELIN Project Briefing 15 AQUAINT Year I Mid-Year Review Sample Results

16. JAVELIN Project Briefing 16 AQUAINT Year I Mid-Year Review Sample Log File Excerpt

17. JAVELIN Project Briefing 17 AQUAINT Year I Mid-Year Review JAVELIN User Interface

18. JAVELIN Project Briefing 18 AQUAINT Year I Mid-Year Review Repository ERD (Entity Relationship Diagram)

19. JAVELIN Project Briefing 19 AQUAINT Year I Mid-Year Review Short-Term Goals Prepare for TREC QA evaluation! Integrate Planner with end-to-end system Acquire Japanese and Chinese resources

20. JAVELIN Project Briefing 20 AQUAINT Year I Mid-Year Review Post-TREC Tasks Execution Manager & UI: –Support for interactive dialog –Extended evaluation capability –Ability to re-run prior question with modifications Repository: –Support for answer justification –Support for end-user repository search

21. JAVELIN Project Briefing 21 AQUAINT Year I Mid-Year Review Post-TREC Tasks [2] Planner: –Ablation studies –Advanced question types –Planning parameter variations Question Analyzer: –Broaden coverage of question parsing –Support for additional question types –Produce interlingua for request object

22. JAVELIN Project Briefing 22 AQUAINT Year I Mid-Year Review Post-TREC Tasks [3] Retrieval Strategist: –Add Japanese and Chinese document collections, relational DB support –Add Google as a document collection –Support for answer verification –Switch to Lemur Toolkit Request Filler: –Reference resolution –Deeper NL analysis

23. JAVELIN Project Briefing 23 AQUAINT Year I Mid-Year Review Post-TREC Tasks [4] Answer Generator: –Combining more evidence types (predicate argument structure, event boundaries) –Extended answer types (hypotheticals)

24. JAVELIN Project Briefing 24 AQUAINT Year I Mid-Year Review Gathering Evidence in Answer Generation Q: Name all the bills that were passed during the Bush administration. Not likely to find passages mentioning `bill’, `pass’, `Bush administration’. When was Bush administration?? `Symbolic’ QA: look for explicit answer in collection, might not be present. `Statistical’ QA: look at distribution of documents mentioning Bush administration. Combining evidence of different sorts!

25. JAVELIN Project Briefing 25 AQUAINT Year I Mid-Year Review Gathering Evidence [2] Can we figure out if Bush administration was around when document was written? Look at tense/aspect/wording. Forward time references –Bush administration will do something Backward time references –Bush administration has done something Hypothesis: –Backward time references provide information about onset of event; –Forward time references provide information about end of event.

26. JAVELIN Project Briefing 26 AQUAINT Year I Mid-Year Review Clustering Evidence Bush administration forward references Administration change Event end Time stamps #docs mentioning Bush adm. on given day

27. JAVELIN Project Briefing 27 AQUAINT Year I Mid-Year Review Clustering Evidence [2] Bush administration backward references #docs mentioning Bush adm. on given day Time stamps Administration change Event onset