1. AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation

2. Testbed Activities Testbed –Provide access to Q&A systems on classified data. –Solicit user feedback and conduct user studies. Testweb –Provide public access to a broad variety of Q&A capabilities. –Evaluate systems and architectures for inclusion in testbed. User Studies –Obtain feedback from users on utility and usability Architecture support

3. Testbed Classified network (ICTESTNET) –access to users, data, scenarios will be restricted Evaluate systems prior to installation –Testweb is a good first step –MITRE installations are more than rehearsal

4. Testbed Progress MITRE installations –QUIRK (CYCorp/IBM) –Javelin (CMU) –LCC Assessing –Portability to the IC environment –Maintainability –Features –Resources –Configuration

5. Deployment Issues Restrictive version requirements Fixed pathname/hostname issues Configure scripts Remote debugging (installation, configuration) is inefficient Differences in machine configuration, environment Unknown or unrealized dependencies Schedule conflicts Firewalls and other security issues

6. Developing a Testbed Vision USNR Analyst Task MITRE gains first-hand knowledge about: –analyst’s working environment tools methods of analysis –nature of analyst’s tasking –long-term analysis goals MITRE obtains expert feedback about: –suitability of QA tools for analysis tasks –representativeness of CNS scenarios –methodology for further tool assessment –methodology for future user studies

7. Plan for USNR Visit Discussions of analyst's history with analysis tasks Hands-on presentation of QA and Search tools MITRE and USNR analyst jointly work through 1 to 2 CNS scenarios using QA tools, search tools, telephone, etc. Analyst formally works through 1 to 3 more CNS scenarios on their own –MITRE monitors the activity through logging and survey instruments –The analyst provides feedback so adjustments can be provided as necessary The analyst and MITRE will jointly analyze the results MITRE and the analyst develop a draft test plan

8. Testweb Q/A system CNS TREC 2002 Javelin, LCC, Qanda, TextMap Q/A Portal/ Demo Q/A repository Google API Other collections IR service Clarity service Users Q/A Portal/ Demo Loose federation of systems Simple services over TCP/HTTP/SOAP

9. Q&A Demo

10. CNS Data http://aquaint.mitre.org/cns

11. UMASS Clarity Many thanks to Steve Cronen-Townsend & Bruce Croft Clarity measures … Clarity service –SOAP interface –Access via Google Java client –Data sets: AQUAINT CNS, TREC-11 (soon) User interface –For demonstration, simple output of clarity score –To come, low clarity warning

12. CNS Search w/ UMASS Clarity http://aquaint.mitre.org/clarity

13. IR Service Utilizing Google SOAP API Tied SOAP to Lucene search Data sets: AQUAINT CNS, TREC-11 Access via Google Java client

14. Q/A Architecture Meeting: April 22, 2003 @ IBM in Hawthorne, NY Eric Nyberg (CMU), David Ferrucci & Chris Welty (IBM), Scott Mardis & John Burger (MITRE) Topics: –Javelin –PIQUANT –Catalyst –UIMA –AQUAINT Testbed

15. Some Directions Architecture is important –Enable collaboration –May be task dependent Common Components/Subsystems –Question analysis –Query formulators –IR –Taggers –Knowledge sources

16. MITRE’s Mission on Architecture Encourage collaborations –Promote regularization of interfaces Facilitate broader use of best components Advocate for users (IC analysts) and user tasks Coordinate with Ex. Comm.

17. MITRE’s Roles on Architecture Services Community Support APIs, Standards Infrastructure Resources Data Resources

18. MITRE’s Roles on Architecture User logs, including Q&A, simple search, documents viewed, etc. Extended WN & XWN APIs Query formulation API Corpora URIs Services, Libraries APIs, Standards Infrastructure Resources Data Resources APIs, Standards: Community Support

19. MITRE’s Roles on Architecture IR Clarity Unit conversions ? Time normalization ? Community Support Infrastructure Resources Data Resources Services, Libraries: APIs, Standards Services, Libraries

20. MITRE’s Roles on Architecture Modified CNS corpus Ontology library ? Community Support Infrastructure Resources Data Resources: APIs, Standards Services, Libraries Data Resources

21. MITRE’s Roles on Architecture Middleware tools ? Infrastructure Resources: APIs, Standards Services, Libraries Data Resources Community Support Infrastructure Resources

22. MITRE’s Roles on Architecture Tool website (index) Wiki for architecture ? Mailing lists Infrastructure Resources Community Support: APIs, Standards Services, Libraries Community Support Data Resources

23. Component Offerings LCC –Ontology tools (development & browsing) –Extended WordNet (XWN) ISI –Question analysis (QTarget) –Question paraphraser –Query formulator –Web-based IR –Answer identifier (extractor)

24. Component Offerings (cont’d) IBM –NE tagger (Resporator – 80 types) CyCorp –Answer sanity checking –Taxonomy (well-developed for CNS/WMD) –Unit conversion –Cyc concept paraphrases UMASS –Clarity

25. Component Needs ? IR subsystems User context & history representation More Ontologies & Tools Answer combination Entity normalization (time, location, organization) Structured data services (DBs) Other media

26. Many Faces of AQUAINT Leverage all of AQUAINT’s successes Expose a greater range of interfaces to analysts –They may find something they can use now –Make the Q/A system more transparent Exploiting the middle ground –Queries that are more complex than boolean search, but more structured than free natural language questions