1. NRRC Summer ‘02 Workshop Proposal: Habitability December 5, 2001 MITRE Christy Doran, MITRE Joe Marks, MERL cdoran@mitre.org marks@merl.com

2. Page 2 Copyright © 2001 The MITRE Corporation. All rights reserved. Problem  Basic premise: Assume an imperfect Q&A system -- what can we do to make it more usable?  Our proposal (courtesy of Bill Ogden, NMSU) is to enhance the habitability of Q&A systems “Habitability, as the term will be used here, is obtained when a significant proportion of the users utterances are interpreted in such a way that the user in some meaningful sense is carried closer towards accomplishing the task at hand.’’ Ivan Bretan citing: W.C. Watt, "Habitability," American Documentation, July, 1968, pp. 338--351.

3. Page 3 Copyright © 2001 The MITRE Corporation. All rights reserved. What might this mean for Q&A systems?  Representing system expertise -With the goal of imbuing the system with an ability to articulate the extent of its knowledge  Presenting query results (or lack thereof) -Communicating data and metadata efficiently  Explaining how the system works -Only an issue when it doesn’t work well?  Incremental question/answer refinement -Human-computer collaboration  Allowing teamwork (human) -Human-human collaboration

4. Page 4 Copyright © 2001 The MITRE Corporation. All rights reserved. Approach  Habitability task not sufficiently well defined at this time -No obvious silver bullet  Interaction needed between at least three different communities: -HCI (Visualization, Groupware) -NLP (Dialogue, Q&A) -Target users  Our proposal: Workshop that would bring these communities together to further explore the topic and identify approaches to improving habitability of Q&A systems

5. Page 5 Copyright © 2001 The MITRE Corporation. All rights reserved. Details of the workshop  2.5-day workshop  Meeting format Day 1 AM: All participants to run and comment on a Q&A system, and to experiment with other representative tools  IBM Q&A system (Roukos & Ittycheriah) Day 1 AM: Historical motivation by Bill Ogden Day 1 PM: Brief presentations from each subarea for establishing common appreciation of issues & challenges Day 2: Structured discussion based on first day’s activities to identify most promising approaches for improving habitability of Q&A systems Day 3 AM: Decide on best way(s) to pursue the approaches identified at a follow-on event

6. Page 6 Copyright © 2001 The MITRE Corporation. All rights reserved.  Goal is 23 participants, including Doran, Marks, and Ogden  Natural language processing (NLP) -Dialogue: Lyn Walker (AT&T), Diane Litman (Pittsburgh), Candy Sidner (MERL), David Traum (USC ICT) -Q&A: James Pustejovsky (LingoMotors), Liz Liddy (Syracuse), Marc Light (MITRE), Sanda M. Harabagiu (SMU)  Human-computer interaction (HCI) -Visualization: Marti Hearst (UC Berkeley), Kent Wittenburg (MERL), Jock Mackinlay (Xerox PARC), Stephen Eick (Visual Insights) -Groupware: Kori Inkpen (Dalhousie), Terry Winograd (Stanford), Mark Ackerman (Michigan), Saul Greenberg (Calgary)  Q&A system users (members of NRRC Specialists Group) -Mark Zimmerman (USGC), John Donelan (USGC), Kelcy Allwein (USGC), Jean-Michel Pomarede (USGC) Proposed workshop participants Bold = preliminary commitment

7. Page 7 Copyright © 2001 The MITRE Corporation. All rights reserved. Results of the workshop  Meeting tentatively scheduled for late February  Workshop to be documented in an article or report or roadmap to engage relevant communities in Q&A research  Another possible outcome is a proposal for a follow-on event, e.g., a AAAI Symposium or a conference workshop

8. Page 8 Copyright © 2001 The MITRE Corporation. All rights reserved. Resources and costs  Workshop -Travel and honoraria: $34K -Site expenses: @MITRE $10K, more compelling location $25K  Lodging and local transportation  Transportation and set-up of demo equipment -Estimated total: cheaper $44K, less cheap $59K  Follow-on activity: $50K -This could be a second workshop, or other activity selected by the group