AQUAINT Testbed Activity John Burger, Marc Light, Scott Mardis, Alex Morgan The MITRE Corporation © 2002, The MITRE Corporation

Models of Technology Transfer Research Transfer Iterative feasibility experiments End of program handoff Parallel Transfer

MITRE’s role in AQUAINT Researchers Deployment User Eval & Analysis User User Testbed EAEA EAEA EAEA EAEA EAEA

What types of things we might learn l Performance constraints - System response needs to be shorter than X, if a search- engine-like interaction is to be used. l Features needed - Questions tend to be answered by only one source and thus source selection could be a win. - Questions involving temporal reasoning are frequent. - Answers tend to occur only once in the sources. l User interface feedback - Users prefer knowing answer source and pedigree. l System internal - Question sequences tend to utilize a small pool of documents. - Tables tend to contain answers to "how many" questions.

Testbed Dimensions Goal: give analysts useful QA capabilities Features & Functionality what question types what answer types User interface speed of technology access technology tool suite vs. system Data feeds vs. static text vs. speech vs. IM structured vs unstruct multi vs. monolingual Research Dimensions Operational Dimensions Users, Domains, Scenarios Plumbing server platform existing clients network protocol Measuring Usefulness instrumentation surveys

Picking the Right Point(s) in the Space l One can start with any one dimension and constrain the others l However, we recommend - Start with users and domain - The data, functionality, and interface dimensions are then greatly constrained - The scenarios constrain the measuring usefulness dimension l Start with “easy” point l Multiple points will be needed

Example: Nuclear Non-Proliferation Features & Functionality question types: ? answer types: ? User interface ? Data unstructured English language reports on non-proliferation, country reports PPNN database mostly static, some live Users IC analysts Domains nuclear non-proliferation Scenarios creating country profiles Plumbing IC-TESTNET mitre.org Measuring Usefulness instrument server Research Dimensions Operational Dimensions

Example: Chemical Transportation Emergency Center Features & Functionality list questions factual questions temporal facts believability of data is NOT an issue User interface ? Data feeds and static text, telephony chem Hazard DB English Research Dimensions Users, Domains, Scenarios CTEC Plumbing no issues with classified data Measuring Usefulness instrumentation surveys Operational Dimensions

Other User Group Possibilities l AQUAINT researchers - news l Air Force Information Warfare Center - computer networks & security l Armed Forces Medical Intelligence Center - biological threats l Focus group from multiple intel agencies - unified by a single topic of interest We will reach a decision on an initial user group by Fall, 2002

Purpose of this demo l Gain some momentum - Experience for MITRE - Experience for AQUAINT contractors l Jump start discussion - What do we really want? l The testbed is not intended for comparing systems - Not a different kind of TREC l Integration is desirable, but not a requirement - Fielding individual system may make sense l User modality is wide open - Browser, chat, newsgroup, ,...

Demo Architecture MITRE Web server LCC QA server MITRE QA server CMU QA server question 1 answers 4 question 2 answers HTTP TCP l QA server connections use ad hoc protocol over TCP l HTTP would simplify network security issues l Next steps: - add more sites - collate answers: merge, cluster, rank - preview to small number of users

Testbed Stages for Developers l QA servers located at development sites - promoting stability - provides access to open source collections - on analysts’ desktop l QA servers located at mitre.org - rehearsal for classified environment - packaging, simplifying distribution - performance improvements? l QA servers located on IC-TestNet - operational data - limited access (classified, non-internet, separate physical lab)

l Classified network - access to users, data, scenarios may be restricted l Available at multiple agencies - potentially accessible by many analyst groups - access is not from the analysts’ primary environment - user labs in the planning and construction stage l No electronic connections to IC-TestNet - occasional updates possible, but static data easier - no remote monitoring or maintenance ! l Requirements page proposal - may need some hardware IC-TestNet Deploy here only after some successes in unclassified environments ! !

Architecture MITRE QA Portal Data capture QA server One way to do it. documents 1 question 3 4 answer 5 6 documents 2 (Every n hours/days)

QA server Architecture MITRE QA Portal IR Engine QA server Another way. question 2 3 answer 6 documents 5 query 4 answer 7 documents 1 QA server IR or KB server

Component Integration l Collaborate with AQUAINT’s full system contractors - share service or share executable - keep us in the loop so that we can compile and promote best practices, and reduce duplicate effort l Integrate with QANDA - download from - wrap module - MITRE will assist

Testbed Schedule l Preliminary user group(s) identifiedAug, 2002 l Identify tasks/scenariosAug, 2002 l Propagate constraintsSept, 2002 l Non-classified MITRE-hosted QA portal - installation Oct, pilot study Dec, 2002 l Classified (IC-TestNet) - proposal Dec, hardware installation Mar, software installation May, pilot study Sept, 2003