1. Outline Grammar-based speech recognition Statistical language model-based recognition Speech Synthesis Dialog Management Natural Language Processing © 2013 by Larson Technical Services1

2. Dialog Management Controlling the interchange of information between users and application Three dialog styles 1.Human-directed conversational dialogs User asks a question or speaks a command and the computer responds. 2.Application-directed conversational dialogs Application asks questions to solicit answers and instructions from a user. 3.Mixed-initiative dialogs User and application take turns driving the conversations. © 2013 by Larson Technical Services2

3. Three Dialog Styles Application-directed Application: What month? Caller: February Application: What day of the month? Caller: Twelve Application: What year? Caller: Nineteen ninety-seven Human-directed Caller: Set month to February Application:Month is February Caller: Set day to month? Application: Day is twelve Caller: Set year to nineteen ninety-seven Application: Year is nineteen ninety-seven © 2013 by Larson Technical Services3 Mixed-initiative Application: What month? Caller: February twelve nineteen ninety-seven

4. VoiceXML 2.1 XML format for specifying interactive voice dialogues between a human and a computer – DTMF input and prerecorded voice as output – Speech recognition and speech synthesis – Video output to user (non-standard) Designed for Interactive Voice Response (IVR) applications using telephone Currently does not support external events, except and Requires a VoiceXML interpreter © 2013 by Larson Technical Services4

5. Example of VoiceXML 2.1 Fragment … Which account savings or checking savings checking CD certificate of deposit $ = “CD” …. … © 2013 by Larson Technical Services5 Dialog Language (VoiceXML 2.1) Speech Synthesis Markup Language (SSML) Speech Recognition Grammar Specification (SRGS) Semantic Interpretation (SI)

6. Example of VoiceXML 2.1 Fragment … Which account savings or checking savings checking CD certificate of deposit $ = “CD” …. … © 2013 by Larson Technical Services6 Dialog Language (VoiceXML 2.1) Speech Synthesis Markup Language (SSML) Speech Recognition Grammar Specification (SRGS) Semantic Interpretation (SI)

7. Example of VoiceXML 2.1 Fragment … Which account savings or checking savings checking CD certificate of deposit $ = “CD” …. … © 2013 by Larson Technical Services7 Dialog Language (VoiceXML 2.1) Speech Synthesis Markup Language (SSML) Speech Recognition Grammar Specification (SRGS) Semantic Interpretation (SI)

8. Example of VoiceXML 2.1 Fragment … Which account savings or checking savings checking CD certificate of deposit $ = "CD" …. … © 2013 by Larson Technical Services8 Dialog Language (VoiceXML 2.1) Speech Synthesis Markup Language (SSML) Speech Recognition Grammar Specification (SRGS) Semantic Interpretation (SI) Text recognized by the speech recognizer is placed into the variable " account "

9. VoiceXML 2.1 Features Menus, forms, sub- dialogs –,, Inputs – Speech recognition – Recording – Keypad Output – Audio files – Text-to-speech Variables –,, © 2013 by Larson Technical Services9 Events –,,,, Transition and submission –, –Telephony –Connection control –, –Telephony information –Platform –Objects –Performance –Fetch

10. VoiceXML 2.1 FIA © 2013 by Larson Technical Services10 Forms Interpretation Algorithm (FIA) selects and processes fields producing an application-directed dialog

11. Visual VoiceXML VoiceXML field -> text field To order please enter your PIN for identification http://www.jacada.com/

12. Other Software for Dialog Management CCXML State Chart XML EMMA Visual VoiceXML Do It Yourself © 2013 by Larson Technical Services12

13. Call Control XML (CCXML) An event processing language – Originally used to manage telephone calls – Now used to process events from outside of VoiceXML (incoming calls, messages from system devices, etc.) – Invokes VoiceXML to interact with user CCXML has no UI features © 2013 by Larson Technical Services13

14. State Chart XML (SCXML) State Chart XML (SCXML): State Machine Notation for Control Abstraction – http://www.w3.org/TR/scxml/ http://www.w3.org/TR/scxml/ © 2013 by Larson Technical Services14

15. EMMA Extensible Multimodal Annotation markup language Canonical structure for semantic interpretations for a variety of inputs including: – Speech – Natural language text – GUI – Ink © 2013 by Larson Technical Services15

16. EMMA Ink Interpretation Speech Recognition Merging/ Unification Speech Ink EMMA Grammar + Semantic Interpretation Instructions Interpretation Instructions Applications © 2013 by Larson Technical Services16

17. Do-It-Your-Self Dialog Management Use a programming or scripting language to specify control Use APIs to access speech technologies – Web Speech API https://dvcs.w3.org/hg/speech-api/raw-file/tip/speechapi.html – The W3C HTML-Speech Incubator Group Final Report: http://www.w3.org/2005/Incubator/htmlspeech/XGR-htmlspeech- 20111206/ http://www.w3.org/2005/Incubator/htmlspeech/XGR-htmlspeech- 20111206/ – Proprietary APIs Google, Microsoft, Apple, etc. © 2013 by Larson Technical Services17

18. Summary: Dialog Management 1.Application-directed conversational dialogs Novice users Specialized grammars for each dialog point 2.Human-directed conversational dialogs Experienced users Dictation and SLMs 3.Mixed-initiative dialogs Allows users to grow from novice to experienced are their own pace Complex to develop © 2013 by Larson Technical Services18