Alicia Abella AT&T Labs – Research Florham Park, New Jersey Semantic Information Processing of Spoken Language - How May I Help You? sm Alicia Abella AT&T Labs – Research Florham Park, New Jersey

Motivation Goal is to provide automated customer services via natural spoken dialog. Natural means what people actually say, rather than what we’d like them to. Shift burden from user to machine

Why Spoken Language Understanding? User interface as the bottleneck to exploiting speech and language processing technological advancement Spoken language as focus of this work Machine Initiative Menus (please say collect, calling card …) Classes (please say credit card number, destination city, person name, date, …) User Initiative How may I help you?

Stroustrup on programs which communicate with people "... it must cope with that person's whims, conventions and seemingly random errors. Trying to force the person to behave in a manner more suitable for the machine is often (rightly) considered offensive.” from "The C++ Programming Language”(1987) pp. 76

A History of Applications Department Store Call-Routing (1989-1991) Almanac Data Retrieval (1992) Airline Travel (1993) Multimodal Blocks World (1993) Operator Services (1995-9) Customer Care (2000+) Enterprise Customers (2002+)

How May I Help You? SM . . . Prompt is “AT&T. How may I help you?” User responds with unconstrained fluent speech System recognizes and determines the meaning of users’ speech, then routes the call Dialog technology enables task completion HMIHY . . . Local Account Balance Calling Plans Unrecognized Number

Extracting Meaning from Speech Extracting meaning is primary in speech understanding systems. How to quantify the information content of a natural language message? Such theory is crucial to engineering devices which understand and act upon such messages.

Communication Paradigm Goal of communication is to induce the machine to perform some action undergo some internal transformation Communication is successful if the machine responds appropriately Contrast with traditional communication theory

Shannon (1948) The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning, … These semantic aspects of communication are irrelevant to the engineering problem.

Architecture for Natural Spoken Dialog Voice reply to customer Speech Speech Text-to-Speech Synthesis Automatic Speech Recognition TTS ASR Data Words to be synthesized Words Words Words spoken Spoken Language Generation SLG SLU Dialogue technologies are the closest to an application Accurate understanding can often be achieved without correctly recognizing every word SLU makes it possible to offer services where the customer can speak naturally without learning a specific set of terms. Provide service differentiation, revenue and cost reduction. UI is very critical in building an application The technology and advancements in SLU and DM are at the state where ASR was over a decade ago. SLU and DM Has only become possible over the past few year when ASR reached real-time performance. You do not need to recognize every word to do accurate spoken language dialogue I remember a comment Jont Allen made once about the woman in this image. I couldn’t have agreed more with him. Can we find a woman who doesn’t look like she’s coming from the 50s? Afterall we’re talking about technology for the 21st century we should show an image that is more “hip” with the times.  Spoken Language Understanding Action DM Meaning Meaning Dialogue Management

Architecture for Natural Spoken Dialog Play prompt ASR Spoken Language Understanding Dialog Manager User speech Acoustic Models Language Models Salient Grammar Fragments Inheritance Hierarchy

Technology Component Traits Robustness ASR Large vocabulary > 10,000 words Dialects (Nationwide deployment) Real-time SLU Tolerance of varied phraseology Many ways of saying the same thing Similar way of saying different things Dialog Confirmation, Re-prompting, Context switching Say anything, anytime, anyway

Examples of Customer Utterances Account Balance Other General Billing Rates and Calling Plans Charge on Bill Change Customer Info

Comparative Example: Unrecognized Number Customer Care IVR HMIHY Sparkle Tone “Thank you for calling AT&T…” Sparkle Tone “AT&T, How may I help you?” Network Menu Account Verification Routine LEC Misdirect Announcement Each “Menu” has 4 – 6 prompts within it. The 0300 example was created off the system working today. It is not the new 8.0 Release that will be deployed in a few weeks. (The main difference is there is an additional menu in the current flow) Additionally this calling party is classified in RAMP as a customer who has some sort of credit problem and therefore receives the prompt about it In this HMIHY demo it is sending the caller to the DTMF version from 0300, the plan is to use the ASR version of 0300 Account Verification Routine Main Menu LD Sub-Menu Reverse Directory Routine

Example Dialogs Rate Plan Account Balance Local Service Unrecognized Number Threshold Billing Billing Credit

The technology is in use today AT&T Customer Care organizations Consumer service live since Nov. 2000 Decreased servicing costs; reduced time spent in automation; reduced repeat calls and customer defections Small Business service pilot underway Supports 800#s used for billing inquiries and corporate calling card transactions. Determines caller intent to perform activities such as making payments, requesting bill adjustments, ordering cards, reporting stolen cards AT&T Enterprise customers Several in beta trials Delivers this functionality in a networked, managed environment.

Industry Specific Applications Check account balances, Apply for mortgage, Request credit report, Locate branch Verify coverage, Inquire about a claim, Check claim status Benefit enrollment, Get a referral, Obtain test results, Pre-admissions procedures Get store hours, Locate nearest store, Directions, Check inventory availability and order status Obtain a price quote, Make reservations, Get a seat assignment, Check flight status, Redeem miles Get instructions, Report a problem, Obtain problem status, Order And Applications Common to All Industries: Password Reset, PIN Reset, FAQs, Help Desk, Locator Services, Order Entry and Status

Key Value Determinants Enhanced Customer Experience Reduced wait and call times even at peak times Natural efficient and personalized dialog Properly fulfills/routes request the first time Decrease Costs Increased use of automation reduces servicing costs Liberate agent headcount Reduce handling time and hang-ups and call-backs Enhanced Business Results Implement new applications to drive revenue No capital investment to build or support Dynamic, customizable resource sharing in secure environment

Observed Benefits Increased automation Improved customer satisfaction Improved routing Ability to add more functionality Improved customer satisfaction Decreased repeat calls (37%) Decreased customer defection rate (18%) Decreased rep time per call (10%) Decreased customer complaints (78%)

Research References at www. research. att. com/~algor/hmihy e. g Research References at www.research.att.com/~algor/hmihy e.g. IEEE Computer Magazine, April 2002