1. Northeast Regional Research Center (NRRC) Summer 2002 Workshops June 11, 2002 MITRE Mark Maybury maybury@mitre.org

2. Page 2 Copyright © 2002 The MITRE Corporation. All rights reserved. Human Science Question Answering  How long it takes the Earth to orbit the sun? One year. 54% correctly answered  Do antibiotics kill viruses as well as bacteria? False. 51%  Human beings developed from earlier species of animals. True, according to the theory of evolution, which is accepted by the majority of scientists, but not by many religious leaders. 53%.  The earliest humans lived at the same time as the dinosaurs. False. Dinosaurs died off millions of years before humans appeared. 48%.  Do lasers work by focusing sound waves? False. Lasers focus light. 45%  The universe began with a huge explosion. True, according to the "Big Bang" theory widely accepted by scientists, but dismissed by some religious leaders. 33% Answer % Correct

3. Page 3 Copyright © 2002 The MITRE Corporation. All rights reserved. Mission and Goals  ARDA -Revolutionary not evolutionary -Well-defined goals, measurable results, scientific methodology  NRRC Goals -Scientific results with + impact on intelligence problems -Engage regional experts (.com,.edu,.org) -Reinforce ARDA thrusts -Infuse technology into government workforce -Transfer technology to and from industry The NRRC is sponsored by ARDA, a US Government entity which sponsors and promotes research of import to the IC which includes but is not limited to the CIA, DIA, NSA, NIMA and NRO.

4. Page 4 Copyright © 2002 The MITRE Corporation. All rights reserved. The NRRC Management Team Executive Director – Dr. Mark Maybury Program Administrator – Bev Nunan Administrative Assistant – Paula MacDonald And other specialists providing infrastructure, subcontracting, security support and technical expertise Penny Lehtola (PM), Rick Steinheiser (COTR) Deputy Program Manager – David Day Program Manager – Penny Chase Speaker Series Administrator – Pam Davis

5. Page 5 Copyright © 2002 The MITRE Corporation. All rights reserved. Executive Committee (EC) Members INDUSTRY  Joe Marks, MERL  Salim Roukos, IBM  Ralph Weischedel, BBN ACADEMIA  Liz Liddy, Syracuse  Stephen Kosslyn, Harvard  George Cybenko, Dartmouth  Gene Charniak, Brown  Bruce Croft, UMass  George Miller, Princeton  Kathy McKeown, Columbia SENIOR GOVERNMENT EXPERTS  Kelcey Allwein, DIA  John Donelan, USGC  Steve Dennis, NSA  Curt Boylls, NSA GOVERNMENT (ex officio)  John Prange, AQUAINT PM  Rick Steinheiser, COTR  Penny Lehtola, ARDA REP PM MITRE (ex officio)  Mark Maybury, NRRC Exec Dir  Penny Chase, NRRC PM

6. Page 6 Copyright © 2002 The MITRE Corporation. All rights reserved. Summer 2002 Workshop  Focus: Advanced QUestion and Answering for INTelligence (AQUAINT) program  Unsolved Q&A Challenges: -Temporal Analysis - Generating a sequence of events along an evolving timeline; resolving temporal references across a series of sources. -Multiple Perspectives - Handling multiple answers to questions from multiple sources with different (e.g., political, operational) perspectives. -Re-use of accumulated knowledge - Maintaining prior knowledge (questions and answers) to help answer the same question later. -Habitability - How can a question & answer system tell a user what it can do and fail gracefully.

7. Page 7 Copyright © 2002 The MITRE Corporation. All rights reserved. Program Committee (PC) Members  Team Leads -James Pusteyovsky (Brandeis) - Temporal -Jan Wiebe (Univ of Pittsburgh) - Multiperspectives -Marc Light (MITRE) and Abraham Ittycheriah (IBM) - Reuse  Technical Experts/Advisors -Ed Hovy (ISI) -Kathy McKeown (Columbia) -Bruce Croft (UMass) -Ralph Weischedel (BBN)  Senior Government Experts -John Donelan, USGC; Steve Dennis, NSA -John Prange, AQUAINT PM  NRRC ExDir, NRRC PL  Ex Officio: ARDA REP PM, COTR

8. Page 8 Copyright © 2002 The MITRE Corporation. All rights reserved. Questions from Web logs  Reuse  could i learn more about the middle ages which is from 500-1500?  Multiple Perspectives -Where Can I find a Dr.'s opinion of Body Piercing? -where can i find opinions about Christmas? -Are my opinions more democratic or republican?  Time -Is there life after death? -Will Jess Powley get her laundry done before the new millenium?

9. Page 9 Copyright © 2002 The MITRE Corporation. All rights reserved. FY02 Current Accomplishments  1 Thrust Area and 4 Focus topics selected  3 Workshops underway (2 Large/1 Small) -participation from industry, academia, and government -4 graduate students, 1 high school student  Major startup issues addressed -proposal process and selection criteria -public release -web site -intellectual property -subcontracting -facilities customized/constructed (3K and 1M) -infrastructure and connectivity  Speaker series launched

10. Page 10 Copyright © 2002 The MITRE Corporation. All rights reserved. Participants  20 from Academia -- Brandeis (3), University of Sheffield, University of Tübingen, University of Michigan (2), Georgetown University, Syracuse University, University of Rochester (2), University of Montreal, University of Pittsburgh (2), SUNY Buffalo, Cornell University (2), Ohio State University, Boston University (2)  11 from Industry -- IBM Research (2), LingoMotors (2), MetaCarta (2), Sun Microsystems, SRA, XEROX PARC, SabIR Research, BBN Technologies  Government -- SPAWAR, plus ARDA & analyst participation  FFRDCs -- MITRE (3)

11. Page 11 Copyright © 2002 The MITRE Corporation. All rights reserved. NRRC Products  LREC QA Workshop Paper, QA Roadmap workshop  Reuse Workshop -Reuse Corpus: 221 single Qs (29 topics), 2130 multiple Qs (13 topics) -100 examples of reuse annotated -Draft Final Report including QA reuse classification  Temporal - TERQAS -TimeML 0.2, annotation guideline 0.1, standards influence -TIMEBANK; algorithms -Report  Multiple Perspectives - MPQA -Perspective conceptualization -Opinion annotation scheme (phrases, sources) -Opinion corpus -Opinion recognition, clustering, evaluation -Report completed in-progress

12. Research Roadmap of Question Answering 2003-2006 Shared data, problems, test suites, evaluation at milestones 2003 2006 Resources (Development, Evaluation) Methodolgies for QA Analysis Resource & Evaluation Methods & Algorithms Systems (Performance & Eval) TIMEBANK Task Modeling Evaluation Measurable Progress Composable Toolkit for QA Productive, Quality QA Resource Selection TIMEML Semi-structured Data QA as Planning Multimodal QA Related Fields: HPKB, TDT, DB, Virtual Ref Desk, User Modeling Constrained QA (Resource/Solution) Feasibility Testing Requirements Determination Empirical Studies USC/ISIS Question Typology TREC QA trec.nist.gov/data/qa.html Collect QA Logs START, FaqFinder, Ionaut, QANDA Web Services (e.g., Google API) Public Taxonomies (e.g., OpenDirectory in RDF) Create QA Sets Multisessional QA (including change detection) Temporal QA QA Reuse Collaborative QA Stereotypical and Indivdualized QA Multilingual QA Interactive Dialog Personalized QA Task Model Reuse across sessions User Typology “Perspective BANK“ Question/Answer Typologies Factoid Questions How Questions Why Questions What If Questions Interoperability Quality Assurance Script/Template Answers Fact Answers Multimodal Answers Answer Toplogy User Expectations Wizard of Oz QA Sets Answer Fusion Answer Justification Crosslingual QA Speed Limit Inference Speed Limit Robust NLP Reusable Test Collection Copyright © 2002 Mark Maybury

13. Page 13 Copyright © 2002 The MITRE Corporation. All rights reserved. Officer asks “Can I see your license?” Top Ten Worst Answers: 10. Sorry, Officer, I didn't realize my radar detector wasn't plugged in. 9. Hey, you must've been doin' about 125 mph to keep up with me. Good job! 8. Do you know why you pulled me over? Okay, just so one of us does. 7. I thought you had to be in relatively good physical condition to be a police officer. 6. Gee, Officer! That's terrific. The last officer only gave me a warning, too! 5. I was trying to keep up with traffic. Yes, I know there are no other cars around. That's how far ahead of me they are. 4. Are You Andy or Barney? 3. Aren't you the guy from the Village People? 2. You're not gonna check the trunk, are you? 1. I can't reach my license unless you hold my beer. (OK in Texas) 12. When the Officer says "Gee Son....Your eyes look red, have you been drinking?" You probably shouldn't respond with,"Gee Officer your eyes look glazed, have you been eating doughnuts?" I pay your salary!

14. Page 14 Copyright © 2002 The MITRE Corporation. All rights reserved. Upcoming Events  June 6 Midterm  June 7 Analytic Disciplines Lecture Day (see nrrc.mitre.org)  July 22 Final Report Out  AAAI Spring SympFuture Directions in Question Answering Monthly Speaker Series at MITRE and via VTC  25 JulyNational Intelligence Analysis and Production John Gannon, former ADCI A&P  9 AugIn-Q-Tel Mr. Gilman G. Louie, President and CEO  17 Sept Technology for Intelligence Dr. John Phillips, Chief Scientist, CIA  22 OctThe Intelligence Analysis Process - Jim Simon, ADCI  17 DecMulti-INT Tasking, Processing, Exploitation, and Dissemination Lt Gen Jim Clapper, USAF (Ret), Dir/NIMA  19 NovMeasurement and Signatures Intelligence Analysis (MASINT) John Morris, former CMO director - SECRET  23 Jan ‘03The Future of Open Source Analysis Gina Genton, Director, CIA/FBIS  23 Feb ‘03Intelligence Analysis Wesley Wark, Fellow, Munk Centre for Int Studies, U. of Toronto

15. Page 15 Copyright © 2002 The MITRE Corporation. All rights reserved. nrrc.mitre.org

16. Page 16 Copyright © 2002 The MITRE Corporation. All rights reserved. Infrastructure  Installed MITRE hardware -15 Linux workstations/laptops -1 ultra-60 server -180 GB shared data space  Data collection examples: LDC Treebank, TIDES/NIST DUC, ACE/NIST EDT, MiTAP, TREC, TDT2, FBIS, …  Lots of applications (e.g., annotation, text processing, ML)  Coming soon: -Linux compute server (2.2GHz Pentium 4; 1 GB RAM) [configuring] -1 Sun Blade 2000 (2 x 900MHz, 2GB RAM) [ordering] -0.42 Terabytes data space [configuring/ordering] -Total of 0.6 Terabytes data space  Plus MITRE’s Research Computing Infrastructure - 300+ machines! completed in-progress