1. Good Morning! How many times have you tried to get ahold of someone and failed?How many times have you tried to get ahold of someone and failed? Need to keep tryingNeed to keep trying Need to find alternate waysNeed to find alternate ways How many times have you forgotten an important appointment?How many times have you forgotten an important appointment?

2. With VERA you can… Schedule wake-ups for yourself and othersSchedule wake-ups for yourself and others Schedule message delivery to yourself and othersSchedule message delivery to yourself and others Schedule recurring callsSchedule recurring calls Send “urgent” messages with return receiptSend “urgent” messages with return receipt Automatically try multiple contacts for a personAutomatically try multiple contacts for a person Automatically try again laterAutomatically try again later

3. Voice Enabled Reminding Agent Vera-OUT: Ula ş Bardak, Scott Judy, Vasco Pedro Vera-IN: Tal Blum, Jeongwoo Ko, Ryosuke Miyata

4. Overview Overall ArchitectureOverall Architecture VeraOUT ArchitectureVeraOUT Architecture SkyperSkyper Dialog Manager InternalsDialog Manager Internals Web interfaceWeb interface DemoDemo Pros and ConsPros and Cons Future WorkFuture Work

5. Overall Vera Architecture Web Interface Vera-OUT** *Vera-IN by Tal Blum, Jeongwoo Ko, and Ryosuke Miyata Vera-IN* DB Skype Internet VoIP Contacts **Vera-OUT by Ulas Bardak, Scott Judy, and Vasco Pedro Skype Switching Center PSTN

6. Vera Server Audio Out Mic Monitor Speakers for User Speech Monitor Speakers for Synthesized Speech Internet Cepstral Speech Synthesizer Sphinx Speech Recongizer Sound Card #1 (Windows Default Audio) Sound Card #2 (Skype Audio) VoIP Skype Software

7. VeraOUT Building Blocks Vera-OUT DM System VeraTM VoIP Skyper VeraDM DB Backend Web Interface Vera-IN Other Components

8. VeraOUT Building Blocks Vera-OUT DM System VeraTM VoIP Skyper VeraDM DB Backend Web Interface Vera-IN Other Components

9. Dialog Management Internals Lang. Understand. PHOENIX/HELIOS Dialog Manag. RavenClaw Lang. Generation ROSETTA Recognition SPHINX Synthesis THETA From Dan Bohus’ talk on Ravenclaw Vera Backend

10. Dialog Management Internals Lang. Understand. PHOENIX/HELIOS Dialog Manag. RavenClaw Vera Backend Lang. Generation ROSETTA HUB Recognition SPHINX Synthesis THETA From Dan Bohus’ talk on Ravenclaw

11. PROCESS MONITOR SPHINX Dialog Management Internals Dialog Manag. RavenClaw Vera Backend Lang. Generation ROSETTA HUB Lang. Understand. PHOENIX/HELIOS Recognition Server Synthesis THETA Multiple, parallel decoders Task Manager VeraTM DB Access Text I/O TTYServer Parsing PHOENIX Confidence HELIOS From Dan Bohus’ talk on Ravenclaw DateTime Skyper

12. Vera-OUT Architecture Vera-OUT DM System VeraTM VoIP Skyper VeraDM DB Backend Web Interface Vera-IN Other Components

13. Database

14. Vera-OUT Building Blocks Vera-OUT DM System VeraTM VoIP Skyper VeraDM DB Backend Web Interface Vera-IN Other Components

15. Web Interface

16. VERA-OUT Demo We showed a demo at this point in the presentation. Here are some recorded sample conversations:

17. RavenClaw Agent Tree E:GetContextA:GetUser A:IsUserR:SpeakWithUserR:WaitForUser A:VeraOut R:Logout I:DeliverMessage A:RightUser E:SetCompleted A:WrongUser I:CallLaterE:CallNext A:DoTask I:WakeUp I:WakeUpSystem E:CheckTaskID I:MessageDeliveredI:MessageNotDelivered I:InformMessageA:messageHandling E:setSuspend R:getUserString E:setRightUser E:setWrongUser E:setNotHere R:WaitForSpeech I:PlayMessage R:getTaskDoneString I:RepeatMessage E:SetMessagePlayed

18. VERA-Out User Tests A user pool of around 10 fully cooperating usersA user pool of around 10 fully cooperating users Used a relatively simpler dialogueUsed a relatively simpler dialogue Problems:Problems: Non-native speakers have a lot of troubleNon-native speakers have a lot of trouble Background noise and booming on the phone can affect the systemBackground noise and booming on the phone can affect the system Speakers get very confused when the system repeats itself with no clarification.Speakers get very confused when the system repeats itself with no clarification.

19. Vera-Out User Test Outcome Increased the dialog cooperationIncreased the dialog cooperation Clarification on what’s being asked forClarification on what’s being asked for Stating what is expectedStating what is expected Added more coverage for speech recognition grammar.Added more coverage for speech recognition grammar. Added more coverage for dialog manager.Added more coverage for dialog manager.

20. VERA-OUT Advantages Skype is inxexpensiveSkype is inxexpensive Skype has worldwide reach.Skype has worldwide reach. No phone line or special equipment required (except for an extra sound card).No phone line or special equipment required (except for an extra sound card). Privacy is preservedPrivacy is preserved No numbers are shared with other users.No numbers are shared with other users. No calls made to a number once the user removes it.No calls made to a number once the user removes it. Ability to call someone at a time when you may not be free yourself.Ability to call someone at a time when you may not be free yourself. PersistencePersistence Ability to confirm delivery of messages.Ability to confirm delivery of messages. Support for recurring tasksSupport for recurring tasks

21. VERA-OUT Issues Skype does cost money when phones are called. What’s the best way to do accounting / charge the right user?Skype does cost money when phones are called. What’s the best way to do accounting / charge the right user? Skype calls show up as “unknown” on caller ID.Skype calls show up as “unknown” on caller ID. Being socially appropriate:Being socially appropriate: Where’s the line between “trying hard enough to make contact” and “harassment”?Where’s the line between “trying hard enough to make contact” and “harassment”? How late/early should VERA call? (time zones?)How late/early should VERA call? (time zones?) Can others override this in an emergency?Can others override this in an emergency? Who decides what an emergency is?Who decides what an emergency is?

22. VERA-OUT Issues (ctd.) Should users be able to cancel all the calls that others have set up for them?Should users be able to cancel all the calls that others have set up for them? Handling failure to make contact.Handling failure to make contact. Calling other contacts.Calling other contacts. Calling back later (after 1 hour)Calling back later (after 1 hour) More work needed.More work needed. Answering machinesAnswering machines How to handle someone who’s not expecting a call from a computerHow to handle someone who’s not expecting a call from a computer

23. VERA-OUT Future Work Better dialog to confirm user identityBetter dialog to confirm user identity Blocked users / Allowed usersBlocked users / Allowed users Calls to multiple usersCalls to multiple users Getting user contacts from the Skype directoryGetting user contacts from the Skype directory Ability to forward and reply to messages.Ability to forward and reply to messages.

24. VERA-OUT Future Work (ctd.) Setting up meetings with multiple participants.Setting up meetings with multiple participants. 1.Call VERA-IN or use the web interface to specify available times and target meeting participants. 2.VERA-OUT calls everyone and finds common times. 3.VERA-OUT gets final approval from the initiator. 4.VERA-OUT calls Roomline to book a room???? 5.VERA-OUT informs everyone of the result.

25. VERA-OUT Conclusion With Vera, reach the people you need to reach, when you want to reach them!

26. Special Thanks! Our professors: Alan Black Alex Rudnicky RavenClaw experts: Dan Bohus Jahanzeb Sherwani Antoine Raux Our longsuffering office-mates: Yee-Man (Betty) Cheng Yanjun Qi

27. VERA-OUT The End