AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation

Testbed Activities Testbed –Provide access to Q&A systems on classified data. –Solicit user feedback and conduct user studies. Testweb –Provide public access to a broad variety of Q&A capabilities. –Evaluate systems and architectures for inclusion in testbed. User Studies –Obtain feedback from users on utility and usability Architecture support

Testbed Classified network (ICTESTNET) –access to users, data, scenarios will be restricted Evaluate systems prior to installation –Testweb is a good first step –MITRE installations are more than rehearsal

Testbed Progress MITRE installations –QUIRK (CYCorp/IBM) –Javelin (CMU) –LCC Assessing –Portability to the IC environment –Maintainability –Features –Resources –Configuration

Deployment Issues Restrictive version requirements Fixed pathname/hostname issues Configure scripts Remote debugging (installation, configuration) is inefficient Differences in machine configuration, environment Unknown or unrealized dependencies Schedule conflicts Firewalls and other security issues

Developing a Testbed Vision USNR Analyst Task MITRE gains first-hand knowledge about: –analyst’s working environment tools methods of analysis –nature of analyst’s tasking –long-term analysis goals MITRE obtains expert feedback about: –suitability of QA tools for analysis tasks –representativeness of CNS scenarios –methodology for further tool assessment –methodology for future user studies

Plan for USNR Visit Discussions of analyst's history with analysis tasks Hands-on presentation of QA and Search tools MITRE and USNR analyst jointly work through 1 to 2 CNS scenarios using QA tools, search tools, telephone, etc. Analyst formally works through 1 to 3 more CNS scenarios on their own –MITRE monitors the activity through logging and survey instruments –The analyst provides feedback so adjustments can be provided as necessary The analyst and MITRE will jointly analyze the results MITRE and the analyst develop a draft test plan

Testweb Q/A system CNS TREC 2002 Javelin, LCC, Qanda, TextMap Q/A Portal/ Demo Q/A repository Google API Other collections IR service Clarity service Users Q/A Portal/ Demo Loose federation of systems Simple services over TCP/HTTP/SOAP

Q&A Demo

CNS Data

UMASS Clarity Many thanks to Steve Cronen-Townsend & Bruce Croft Clarity measures … Clarity service –SOAP interface –Access via Google Java client –Data sets: AQUAINT CNS, TREC-11 (soon) User interface –For demonstration, simple output of clarity score –To come, low clarity warning

CNS Search w/ UMASS Clarity

IR Service Utilizing Google SOAP API Tied SOAP to Lucene search Data sets: AQUAINT CNS, TREC-11 Access via Google Java client

Q/A Architecture Meeting: April 22, IBM in Hawthorne, NY Eric Nyberg (CMU), David Ferrucci & Chris Welty (IBM), Scott Mardis & John Burger (MITRE) Topics: –Javelin –PIQUANT –Catalyst –UIMA –AQUAINT Testbed

Some Directions Architecture is important –Enable collaboration –May be task dependent Common Components/Subsystems –Question analysis –Query formulators –IR –Taggers –Knowledge sources

MITRE’s Mission on Architecture Encourage collaborations –Promote regularization of interfaces Facilitate broader use of best components Advocate for users (IC analysts) and user tasks Coordinate with Ex. Comm.

MITRE’s Roles on Architecture Services Community Support APIs, Standards Infrastructure Resources Data Resources

MITRE’s Roles on Architecture User logs, including Q&A, simple search, documents viewed, etc. Extended WN & XWN APIs Query formulation API Corpora URIs Services, Libraries APIs, Standards Infrastructure Resources Data Resources APIs, Standards: Community Support

MITRE’s Roles on Architecture IR Clarity Unit conversions ? Time normalization ? Community Support Infrastructure Resources Data Resources Services, Libraries: APIs, Standards Services, Libraries

MITRE’s Roles on Architecture Modified CNS corpus Ontology library ? Community Support Infrastructure Resources Data Resources: APIs, Standards Services, Libraries Data Resources

MITRE’s Roles on Architecture Middleware tools ? Infrastructure Resources: APIs, Standards Services, Libraries Data Resources Community Support Infrastructure Resources

MITRE’s Roles on Architecture Tool website (index) Wiki for architecture ? Mailing lists Infrastructure Resources Community Support: APIs, Standards Services, Libraries Community Support Data Resources

Component Offerings LCC –Ontology tools (development & browsing) –Extended WordNet (XWN) ISI –Question analysis (QTarget) –Question paraphraser –Query formulator –Web-based IR –Answer identifier (extractor)

Component Offerings (cont’d) IBM –NE tagger (Resporator – 80 types) CyCorp –Answer sanity checking –Taxonomy (well-developed for CNS/WMD) –Unit conversion –Cyc concept paraphrases UMASS –Clarity

Component Needs ? IR subsystems User context & history representation More Ontologies & Tools Answer combination Entity normalization (time, location, organization) Structured data services (DBs) Other media

Many Faces of AQUAINT Leverage all of AQUAINT’s successes Expose a greater range of interfaces to analysts –They may find something they can use now –Make the Q/A system more transparent Exploiting the middle ground –Queries that are more complex than boolean search, but more structured than free natural language questions