Chatbots Testing Automated testing for Chatbots Jan, 2018

Rise of Chatbots The new bot trend is due to two main factors: artificial intelligence and the rise of messaging applications. “People are now spending more time in messaging apps than in social media and this is a huge turning point.” “Messaging apps are the platforms of the future and bots will provide users with all sorts of services.”  - Business Insider Rise of Messaging Apps

Bot Ecosystem Key Components in the bot architecture are Chat apps, NLP layer & Enterprise Backend Enterprise database to seed Chatbot with the information to respond to customer questions 3 Data store captures data on users’ activities, hits & misses in Chatbot responses 4 Chat apps like Facebook Messenger, Slack, etc. are the interface through which users access and interact with your bot 2 Natural Language Processing translates users’ free form questions into information that can be mapped in real-time to formulate appropriate responses 1

Trending Chatbot Techstack Chatbot platforms Chat messengers Chatbot Platforms backed by Tech Giants Api.ai (Google) Wit.ai (Facebook) LUIS (Microsoft) Watson (IBM) Lex (Amazon) Non programming Chat bot platforms Chatfuel ManyChat Octane Ai Massively Motion.ai.

Testing tenets in Chatbots Testing zones Key testing areas in the bot architecture are highlighted in the below schema DB Tests API Tests Analytics pattern checks Intents & Utterance tests Chatapp Voice, Video and Text validations Natural Language Understanding (NLU) Tests Bot Interoperability testing with messengers

Chatbots Testing Approach 1 Voice & NLP text Testing API Testing UX Testing 2 3 Provide multi accent user intent in the form of Voice/ text to Chatbot and verify its capabilities to process the required ‘Intents’ and respond with the appropriate ‘Utterances’ Use Automated Text to Speech and Natural Language Generators to create combinations of user input & validate the correctness of the Bot responses Understand API Descriptions and endpoints between the Bot UI, NLP Layer and the Backend All combinations of Straight-line cases, boundary cases, null input, Idempotent verbs can be automated and tested at the API level Use Built in assertions in API testing tools like Postman, SoapUI to verify the responses from various layers Assess 3rd Party API Dependencies For Vulnerabilities and perform security testing Achieve Virtualization of backend for faster testing Perform Interoperability testing of Bot with multiple chat messengers and mobile devices(OS version, Device manufactures, form factors) Perform User experience and usability testing of the chat application Measure Mean time/loops to complete transaction and the number of chats to achieve end result Perform NFR testing in targeted mobile and tablets

DAVO - Test Automation framework for Chatbots DAVO Framework 1-Turn Chat Multi- turn chat Automation Libraries Lex API Endpoint Chat UI Utterance Generator Utterances for Intents Utterances with slots DAVO Chat UI Input Text … Enterprise Services

Advantages of DAVO Framework Cross Platform Single Script supports multi platforms like Amazon, FB, Slack Crowd Enabled 360° Test Automation E2E framework to test all layers of Chat, voice, API and Backend Seamless Integration with Crowd platform for exploratory testing Assured CX Voice/Chat app tested with bulk natural user utterances for a successful production Day 1 QA History Shorter Cycle time Maintains HTML Based reports for all chat utterances High reproducibility and Accelerated Feedback to developers for skill tweaks

Lex Chatbot implementation for a Large American Insurance Provider Client is a North America based Insurance company who is one of the US largest insurers of vehicles, homes and small businesses and provide a wide range of other insurance and financial services products with annual revenue of $19B and have 20,000+ employees. Business Problem – The project objective is for Chatbot to deliver canned comments such as initiating, Idle time & ending the chat, guidance to agents on some transactions & answers to FAQ questions there by enabling a quicker response to agent’s inquiries. Customer Situation Farmers is looking at improving the agent experience by reducing hold time for agent servicing through chat and reduce overall costs of call center operation.  Cognizant team analyzed the call center data and determined reasons for agent calls/conversations. Cognizant proposed Amazon Lex as an option to automate the chat conversations. Amazon Lex is a service for building conversational interfaces using voice and text. Lex‘s automatic speech recognition (ASR) and natural language processing/understanding (NLU) capabilities allows you to create self-service experiences. Business Benefits Reduction in call and chat volume to the Service Center Solution Details Intent Keywords are provided as Inputs ( FAQ’s, status of Document uploaded / Policy & Claims). User sessions are to be short and exact the need. When the Chatbot doesn’t able to resolve the customer's issue, Amazon Lex is able to pass on the full context of the conversation to an available CSR representative which keeps the customer from wasting time repeating answers to questions and lets the CSR representative focus 100% of their time on solving the problem, which increases the odds the customer is going complete the contact feeling positive about the experience. Effort Saving up to 30% where there is no need for CSR to be available all the time for Customer initial queries. Different way of responses are been designed and tested like following:- No response — for instance, if someone is idle for sometime. An invalid response — the wrong message entirely not relevant to the chat. A valid response with invalid data —  the correct container of query but with an exclusive word. A valid response — any of a number of variations. Bot only Chat  95% time savings for CSR Bot + CSR Chat  5% time savings for CSR CSR only Chat  0.05% time savings for CSR Client Name: Farmers Insurance NPS: A metric to measure customer feedback GWP: Total premium written and assumed by an insurer before deductions Combined Ratio: (Incurred Losses + Expenses)/Earned Premium