Spoken Dialog Management for an Astronaut’s Procedure Assistant Presented by: Dan Bohus Collaborators: Gregory Aist, RIALIST Group NASA Ames/RIACS Summer Student Research Program August 2002 S.D.M.A.P.A.
Dialog Management for an Intelligent Procedure Assistant2 RIALIST NASA Ames Location: NASA Ames / Moffett Field / Mountain View, CA (CMU West) People Jim Hieronymous John Dowding Beth-Ann Hockey Greg Aist + 7 summer students Projects Personal Satellite Assistant Mobile Robots Intelligent Procedure Assistant
Dialog Management for an Intelligent Procedure Assistant3 Motivation Crew members on International Space Station (ISS) have to perform a wide variety of procedural tasks: ISS maintenance and repair Operations (i.e. EVA Suit Checkout) Setup and running scientific experiments Medical Exams
Dialog Management for an Intelligent Procedure Assistant4 Typical setup Paper documentation 2 or more astronauts involved: one reads, one executes the procedure
Dialog Management for an Intelligent Procedure Assistant5 Proposed alternative Build a spoken dialog system for procedural task guidance and support Create more crew-time! Used also for ground training Potential for increased safety
Dialog Management for an Intelligent Procedure Assistant6 Outline Motivation The Intelligent Procedure Assistant A Dialog Management Component for IPA Demo Future work
Dialog Management for an Intelligent Procedure Assistant7 The Intelligent Procedure Assistant Multi-modal spoken dialog system providing crew support and training for the execution of procedural tasks Guide astronauts through procedural tasks Provide information access (diagrams, docs) Act as a training tool Facilitate task delegation Provide support for voice annotations, intercom
Dialog Management for an Intelligent Procedure Assistant8 IPA Version 0: June 2002 RIALIST IPA Version 0 Load procedure Next Previous Drill-down Describe remaining steps Increase volume Decrease volume Say again
Dialog Management for an Intelligent Procedure Assistant9 Outline Motivation The Intelligent Procedure Assistant A Dialog Management Component for the IPA Demo Future work
Dialog Management for an Intelligent Procedure Assistant10 Speech Recognition Dialog Management Language Generation Synthesis U DSI SONLS Semanticizer (parsing) Resources/Agents Contributions: Dialog Management & Resources
Dialog Management for an Intelligent Procedure Assistant11 Nuance [recognition] Dialog Management Gemini [generation] Festival U DSI SONLS Gemini [parsing] Resources/Agents Nuance [recognition] Dialog Management Gemini [generation] Festival U DSI SONLS Gemini [parsing] Resources/Agents Audio Dialog Agent RavenClaw DM framework (CMU) Contributions: Dialog Management & Resources CORE/ Conversational Skills Dialog Task Specification The actual domain task (IPA domain) Handles dialog flow, initiative, focus shifts, universal dialog mechanisms
Dialog Management for an Intelligent Procedure Assistant12 1. Resources Procedures represented in XML (DTD, XSLT) Title, Description. Steps: Basic Steps, Conditional Steps, Messages, Images Messages: Notes, Cautions Images: Sub-images Water Sampling Procedure 28 steps, 150 sub-steps, 18 conditional steps, 1 image with 5 sub-images HTML version (using XSLT) HTML version
Dialog Management for an Intelligent Procedure Assistant13 2. Dialog Task Development In RavenClaw, a dialog task is implemented as a hierarchy of dialog agents (C++ classes) IPA Welcome LoadProcedureDo Procedures DoProcedure1 IntroClosing DoProcedure2 Step1 Step28Image1 SubStep1 SubStep4 Help Suspend/ Resume
Dialog Management for an Intelligent Procedure Assistant14 2. Dialog Task Development (cont) 52 dialog task agents (types) were implemented: Loading procedures by name Guidance Step-by-step execution Drill-down Branching Presenting notes / cautions / images Information access Talking on/about images, objects Navigational behavior History / Trace through procedure Next, Previous step/substep Start over
Dialog Management for an Intelligent Procedure Assistant15 3. Conversational Skills Enhancements Timeouts System reacts after a timeout period User Barge-in Requests for repeat Requests for help generic “help!” “what can I say ?” Context establishment “where are we ?”
Dialog Management for an Intelligent Procedure Assistant16 Outline Motivation The Intelligent Procedure Assistant A Dialog Management Component for IPA Demo Future work
Dialog Management for an Intelligent Procedure Assistant17 Demo Guidance Volume control Information Access
Dialog Management for an Intelligent Procedure Assistant18 Outline Motivation The Intelligent Procedure Assistant A Dialog Management Component for IPA Demo Future work IPA System (RIALIST Group) Advances in dialog management
Dialog Management for an Intelligent Procedure Assistant19 Future work for IPA system (RIALIST Group) IPA system functionality goals Guide astronauts through procedural tasks Provide information access Act as a training tool Facilitate task delegation Provide support for voice annotations, intercom Iterative improvement process Other features, including those suggested by JSC training personnel and astronauts
Dialog Management for an Intelligent Procedure Assistant20 Timeline … Reliable prototype by the end of September Guidance/Information Access First fieldable system by March 2003
Dialog Management for an Intelligent Procedure Assistant21 NASA is cool !
Dialog Management for an Intelligent Procedure Assistant22 Being back is also cool ! 2 new systems to be developed based on the Communicator SDS setup / RavenClaw DM RoomLine system Provide conference room reservation services within SCS Ridy Lie BusLine system Provide access to bus line information Project “Let’s go!” (Maxine/Alan)? Brian Langner, Antoine Raux Hope you can call them sometime before Christmas !!!