VoiceXML and Internet Telephony Kundan Singh and Henning Schulzrinne Columbia University Joint work (in progress) with Daniel, Naho, Visda and Sean.

18 April, 2001VoiceXML/Kundan Singh/Columbia University 2 Overview A language for specifying voice dialogs in interactive voice response systems Information retrieval –News, sports, traffic, stock quotes e-business –Customer service, banking, stock trading Notification service

18 April, 2001VoiceXML/Kundan Singh/Columbia University 3 PSTN based IVR Platform PSTN End user IVR platform IVR 1 platform Voice and telephony functions (ASR 2, TTS 3, DTMF 4 ) Service logic (application specific) Receives incoming PSTN 5 call Responds back with prompts Accepts user input (DTMF or speech) Takes action based on user input (Usually the service logic is programmed for the specific application, say weather report) [1] Interactive voice response [2] Automated speech recognition [3] Text to speech [4] Dual tone multi-frequency (touch tone) [5] Public switched telephone network Welcome to voice mail. Press 3 to listen to new messages

18 April, 2001VoiceXML/Kundan Singh/Columbia University 4 Decomposition PSTN End user IVR platform Voice and telephony functions (ASR, TTS, DTMF) Service logic (application specific) End user Voice gateway Voice and telephony functions Internet Web server Service logic

18 April, 2001VoiceXML/Kundan Singh/Columbia University 5 VoiceXML PSTN End user Internet Voice gateway Web server Service logic (CGI, servlet, JSP) Voice and telephony functions VoiceXML browser End user VXML HTML DB Multimedia Audio/ grammar Scripts Web server

18 April, 2001VoiceXML/Kundan Singh/Columbia University 6 Why VoiceXML Alternative: write C/C++ application on telephony platforms ? Separate application specific service logic (HTML, VoiceXML) and User interaction (browser, IO device) Can use existing web development tools Can have single application for both web and voice Can use existing infrastructure: HTTP, web servers, etc. Programming voice services for telephony platforms

18 April, 2001VoiceXML/Kundan Singh/Columbia University 7 VoiceXML vs HTML Phone vs PC; IO  phone Transport: HTTP Voice browser vs web browser VoiceXML vs HTML form Enter your Id: Your ID, please.

18 April, 2001VoiceXML/Kundan Singh/Columbia University 8 VoiceXML examples [ 1 ] Hello, World!

18 April, 2001VoiceXML/Kundan Singh/Columbia University 9 VoiceXML examples [ 2 ] Welcome to the weather information service. What state? <grammar src=“state.gram” type=“application/x-jsgf”/> Please speak the state for which you want the weather.

18 April, 2001VoiceXML/Kundan Singh/Columbia University 10 VoiceXML examples [ 2 ] What city? <grammar src=“city.gram” type=“application/x-jsgf”/> Please speak the state for which you want the weather. <submit next=“/servet/weather” namelist=“city state”/> Grammar (city.gram): California | Illinois | New Jersey | New York

18 April, 2001VoiceXML/Kundan Singh/Columbia University 11 VoiceXML examples [ 3 ] … visa {visa} | master [card] {mastercard} | amex {amex} | american [express] {amex} Please say Visa, Mastercard, or American Express. …

18 April, 2001VoiceXML/Kundan Singh/Columbia University 12 VoiceXML examples [ 4 ] Would you like Coffee, Tea, Milk or Nothing. coffee tea milk nothing

18 April, 2001VoiceXML/Kundan Singh/Columbia University 13 VoiceXML examples [ 5 ] Would you like Coffee, Tea, Milk or Nothing. coffee tea milk nothing I did not understand what you said. Please say one of coffee, tea, milk or nothing You must say something. Alternatively: “Would you like ”

18 April, 2001VoiceXML/Kundan Singh/Columbia University 14 Form Interpretation Algorithm Initialize variables, counters. Main loop –Select phase: select next form –Collect phase: prompt and collect input –Process phase: process the event Document: collection of forms An application can use multiple documents

18 April, 2001VoiceXML/Kundan Singh/Columbia University 15 VoiceXML scope Human-Machine Interaction –Audio output (TTS, pre-recorded file) –Audio input (Speech recognition, audio recording) –Character input (DTMF) –Presentation logic (scripting) Basic Connection Control –disconnect –transfer

18 April, 2001VoiceXML/Kundan Singh/Columbia University 16 Application scope General service logic State management Dialog generation Dialog sequencing Database operation

18 April, 2001VoiceXML/Kundan Singh/Columbia University 17 VoiceXML features Menus, Forms, Sub-Dialogs Inputs (grammar, record, dtmf) Outputs (audio, text-to-speech) Events (error handling: nomatch, noinput, catch-throw) Variables and scripting (var, assign, if) Transition or links (goto, submit) Transfer to 3rd party (also add third party) Disconnect the call Platform specific object, and property Pre-fetching

18 April, 2001VoiceXML/Kundan Singh/Columbia University 18 VoiceXML 1.0 VoiceXML 1.0 assign, audio, block, break, catch, choice, clear, disconnect, div, dtmf, else, elseif, emp, enumerate, error, exit, field, filled, form, goto, grammar, help, if, initial, link, menu, meta, noinput, nomatch, object, option, param, property, pros, record, reprompt, return, sayas, script, subdialog, submit, throw, transfer, value, var, vxml Telephony, Speech Synthesis or audio output, User input and Grammar, Program flow, Variable and properties, Error handling, Misc.

18 April, 2001VoiceXML/Kundan Singh/Columbia University 19 Internet Telephony PSTN Internet End user Voice gateway Web server Service logic (CGI, servlet, JSP) Voice and telephony function VoiceXML browser

18 April, 2001VoiceXML/Kundan Singh/Columbia University 20 Internet Telephony PSTN End user SIP user agent Voice gateway Web server CGI, servlet, JSP PSTN/SIP VoiceXML browser with SIP SIP phone New module

18 April, 2001VoiceXML/Kundan Singh/Columbia University 21 Internet Telephony Web server (CGI, servlet, JSP) Example: by phone, voic by phone, directory services for department, web browsing by phone (Not WAP), … VoiceXML VoiceXML browser with SIP SIP phone Accept SIP connection Fetch XML page over HTTP Parse XML Interpret VoiceXML tags Do Text-to-speech Receive and detect user input (DTMF, or in future speech) Parse according to the grammer Fetch audio file from web and play to the user... gateway SIP for signaling, RTP for audio, DTMF (either in- band audio tones or RFC2833)

18 April, 2001VoiceXML/Kundan Singh/Columbia University 22 Status by phone (using TellMe voice browser) Voice XML browser - on going