Convergys Confidential and Proprietary Copyright ©2007 Convergys Corporation Using Excessive Classes To Build a NLU-FAQ System Dhananjay Bansal Nancy Gardner

Convergys Confidential and Proprietary Agenda Introduction Domain Analysis NLU – FAQ approach Results Conclusion

Convergys Confidential and Proprietary Introduction Reason for call can be numerous (can be more then 1000)  Especially true for main telephone numbers Menu driven systems are insufficient to meet customer needs Cumulative loss of containment could be significant. A significant portion of transferred calls are “informational”  More than 35% (domain dependent) How do we serve these callers ?

Convergys Confidential and Proprietary Domain Analysis Large number of major classes (> 500) with relatively fewer sub classes (< 6) for a major class Most HMIHY responses identify the major class Transcription data  Recordings of actual responses to HMIHY question in the application  Large number of transcription and annotation needed  Data sparseness issues Website a good source for information  Websites usually has more information then IVR  Significant cumulative information  Information can be used in IVR Goal: Be able to recognize most classes (> 500)

Convergys Confidential and Proprietary An Example Website Map

Convergys Confidential and Proprietary Example Leaf Webpage How to apply  Continue with application  Phone: (nnn) nnn-nnnn, Monday through Friday, 8:00am to 6:00pm ET  In person: find a Branch or Securities Office near you  By mail: o Complete and sign the Roth IRA Plan Application and read the Roth IRA Fact Book o Return completed forms, along with contributions, to: What you need to apply You will need the following pieces of information:  Your Social Security Number (if applicable)  Beneficiary's Social Security Number and Date of Birth  If you are rolling over an employer-sponsored plan, you will need your plan provider/administrator's name and contact information. What to expect when applying online  The application will take approximately 20 minutes to complete.  Within two business days after we receive your application, a Retirement Specialist will contact you to discuss your retirement needs.  After speaking with a Retirement Specialist, final account forms will be sent for you to sign and return.

Convergys Confidential and Proprietary Use NLU to identify Major Class  Follow up with Menu Driven systems to narrow down choices NLU-FAQ Design

Convergys Confidential and Proprietary NLU Grammar Development Each “Class” corresponds to a leaf (information) webpage Use contents in the website, links, web headers of a content page and the parent pointers to generate data  Web crawler, multiple parent paths Merge the actual call recording transcriptions Generalize the data to include variations, prefixes, suffixes. Improve with tuning

Convergys Confidential and Proprietary Content Development Disambiguation prompts for “Major” classes  Do you need A, B, or C, Or Say Customer Service.  FAQ Content for each Class. Classes and contents manually reviewed and developed.

Convergys Confidential and Proprietary Results Correct recognition accuracy of major classes Expected containment around 5%

Convergys Confidential and Proprietary Discussion How to effectively contain and serve low frequency, but numerous call reasons Data Generation: How to better formulate questions from answers Recognition/Classification Accuracy NLU-FAQ system is 1 of many ways The approach can be deployed across multiple verticals

Convergys Confidential and Proprietary Copyright ©2007 Convergys Corporation Using Excessive Classes To Build a NLU-FAQ System Dhananjay Bansal Nancy Gardner