A multimodal dialogue-driven interface for accessing the content of recorded meetings Agnes Lisowska ISSCO/TIM/ETI University of Geneva IM2.MDM Work done in collaboration with: Martin Rajman, Mirek Melichar, Trung H. Bui (LIA & CGC at the EPFL) and Marcin Bogobowicz (graphic artist)

The Problem The data Audio and video Documents –printed documents –slides –whiteboard activity –handwritten notes Annotated transcriptions –topics –keywords –argumentative structure The users Manager –tracking project progress –tracking employee performance Employee –catch up on a missed meeting –check facts –learn about a project What type of interface?

User-Centered Design (UCD) Which functionalities? Which modalities? Which combinations? Degree of user control? Type of dialogue strategy? Data organization Data annotation Types of data What tasks will users be doing? Who will the users be? “The bottom line is that the tool should be designed around the needs of the user to allow for effective and efficient use of the application.” (Scanlon and Percival, IBM)

User Requirements Requirements gathering must be an iterative process –requirements can emerge or change over time and with experience User requirements gathering within IM2.MDM –Questionnaire-based study (Lisowska 2003) collected ~300 queries to the system analyzed linguistically and thematically results used to derive user requirements and drive preliminary design of Archivus interface

HCI and Design Principles in Archivus 1.Interaction metaphor –helps a user understand the various functionalities of a system 2.Multiple levels of interaction (user experience differs) 3.Visibility –important and frequently used elements always visible 4.Consistency –same elements in same location –same functionality performed in same way 5.Feedback to the user –user should see the results of their actions User drivenSystem driven (dialogue system) Mixed initiative (dialogue system + user control) Expert userAverage userNovice user

The Archivus System Designed based on user requirements Interaction metaphor: using a library or archive –library is the database of meetings –each book in the library is a single meeting Preliminary version allows for –many different modalities –full range of interaction styles Modalities –input natural language –speech via speech recognition –text via keyboard or pen pointing –mouse –touchscreen –pen –output audio –system prompts –meeting recordings video –meeting recordings text graphics

The Archivus System Constant view of database (search space) Interactive search definition buttons Interaction history Natural language input feedback System prompts Meeting ‘book’ System control buttons Access to multimedia Notebook

Demo Problem: Denis told you that there was a discussion about red sofas for the ISSCO lounge and you want to know more about it. All you know is: Denis was at the meeting the meeting was about ISSCO lounge furniture the meeting was in early April in Martigny

Testing and Evaluation Wizard of Oz methodology –allows testing without full implementation –needs to be iterative –reveals how modalities are used new user requirements flaws in the design

References & Associated Papers User-Centered Design Scanlon, J & Percival, L. UCD for different project types (Parts 1 & 2). IBM developerWorks website. www-106.ibm.com/developerworks/web/library/us-ucd/ User Requirements Studies Lisowska A., Popescu-Belis A. & Armstrong S. (2004) - User Query Analysis for the Specification and Evaluation of a Dialogue Processing and Retrieval System. LREC 2004 (Fourth International Conference on Language Resources and Evaluation), Lisbon, Portugal, vol.III, p Lisowska, A. (2003) - Multimodal Interface Design for the Multimodal Meeting Domain: Preliminary Indications from a Query Analysis Study. Project Report IM2.MDM-11, November 2003, 30 pages. The Archivus Interface Lisowska A., Rajman M., & Bui T.H. (2004) - ARCHIVUS : A System for Accesssing the Content of Recorded Multimodal Meetings. Proceedings of the Joint AMI/PASCAL/IM2/M4 Workshop on Multimodal Interaction and Related Machine Learning Algorithms, Martigny, Switzerland, 8 p. Dialogue Management T. H. Bui, M. Rajman and M. Melichar. Rapid Dialogue Prototyping Methodology. Proceedings of the 7th International Conference on Text, Speech Dialogue (TSD 2004), P. Sojka, I. Kopecek K. Pala (Eds.), Lecture Notes in Computer Science, Springer-Verlag, Berlin Heidelberg New York, Brno, Czech Republic, September 8-11, 2004 T. H. Bui and M. Rajman. Rapid Dialogue Prototyping Methodology. Technical Report No , Swiss Federal Institute of Technology (EPFL), Lausanne (Switzerland), January, 2004.