1. Chatbots Testing
Automated testing for Chatbots
Jan, 2018

2. Rise of Chatbots
The new bot trend is due to two main factors: artificial intelligence and the rise of messaging applications.
“People are now spending more time in messaging apps than in social media and this is a huge turning point.”
“Messaging apps are the platforms of the future and bots will provide users with all sorts of services.”  - Business Insider
Rise of Messaging Apps

3. Bot Ecosystem
Key Components in the bot architecture are Chat apps, NLP layer & Enterprise Backend
Enterprise database to seed Chatbot with the information to respond to customer questions 3
Data store captures data on users’ activities, hits & misses in Chatbot responses 4
Chat apps like Facebook Messenger, Slack, etc. are the interface through which users access and interact with your bot 2
Natural Language Processing translates users’ free form questions into information that can be mapped in real-time to formulate appropriate responses 1

4. Trending Chatbot Techstack
Chatbot platforms
Chat messengers
Chatbot Platforms backed by Tech Giants
Api.ai (Google)
Wit.ai (Facebook)
LUIS (Microsoft)
Watson (IBM)
Lex (Amazon)
Non programming Chat bot platforms
Chatfuel
ManyChat
Octane Ai
Massively
Motion.ai.

5. Testing tenets in Chatbots
Testing zones
Key testing areas in the bot architecture are highlighted in the below schema
DB Tests
API Tests
Analytics pattern checks
Intents & Utterance tests
Chatapp Voice, Video and Text validations
Natural Language Understanding (NLU) Tests
Bot Interoperability testing with messengers

6. Chatbots Testing Approach1
Voice & NLP text Testing
API Testing
UX Testing 2 3
Provide multi accent user intent in the form of Voice/ text to Chatbot and verify its capabilities to process the required ‘Intents’ and respond with the appropriate ‘Utterances’
Use Automated Text to Speech and Natural Language Generators to create combinations of user input & validate the correctness of the Bot responses
Understand API Descriptions and endpoints between the Bot UI, NLP Layer and the Backend
All combinations of Straight-line cases, boundary cases, null input, Idempotent verbs can be automated and tested at the API level
Use Built in assertions in API testing tools like Postman, SoapUI to verify the responses from various layers
Assess 3rd Party API Dependencies For Vulnerabilities and perform security testing
Achieve Virtualization of backend for faster testing
Perform Interoperability testing of Bot with multiple chat messengers and mobile devices(OS version, Device manufactures, form factors)
Perform User experience and usability testing of the chat application
Measure Mean time/loops to complete transaction and the number of chats to achieve end result
Perform NFR testing in targeted mobile and tablets

7. DAVO - Test Automation framework for Chatbots
DAVO Framework
1-Turn Chat
Multi- turn chat
Automation Libraries
Lex API Endpoint
Chat UI
Utterance Generator
Utterances for Intents
Utterances with slots
DAVO Chat UI
Input Text …
Enterprise Services

8. Advantages of DAVO Framework
Cross Platform
Single Script supports multi platforms like Amazon, FB, Slack
Crowd Enabled
360° Test Automation
E2E framework to test all layers of Chat, voice, API and Backend
Seamless Integration with Crowd platform for exploratory testing
Assured CX
Voice/Chat app tested with bulk natural user utterances for a successful production Day 1
QA History
Shorter Cycle time
Maintains HTML Based reports for all chat utterances
High reproducibility and Accelerated Feedback to developers for skill tweaks

9. Lex Chatbot implementation for a Large American Insurance Provider
Client is a North America based Insurance company who is one of the US largest insurers of vehicles, homes and small businesses and provide a wide range of other insurance and financial services products with annual revenue of $19B and have 20,000+ employees.
Business Problem – The project objective is for Chatbot to deliver canned comments such as initiating, Idle time & ending the chat, guidance to agents on some transactions & answers to FAQ questions there by enabling a quicker response to agent’s inquiries.
Customer Situation
Farmers is looking at improving the agent experience by reducing hold time for agent servicing through chat and reduce overall costs of call center operation.  Cognizant team analyzed the call center data and determined reasons for agent calls/conversations. Cognizant proposed Amazon Lex as an option to automate the chat conversations. Amazon Lex is a service for building conversational interfaces using voice and text. Lex‘s automatic speech recognition (ASR) and natural language processing/understanding (NLU) capabilities allows you to create self-service experiences.
Business Benefits
Reduction in call and chat volume to the Service Center
Solution Details
Intent Keywords are provided as Inputs ( FAQ’s, status of Document uploaded / Policy & Claims).
User sessions are to be short and exact the need.
When the Chatbot doesn’t able to resolve the customer's issue, Amazon Lex is able to pass on the full context of the conversation to an available CSR representative which keeps the customer from wasting time repeating answers to questions and lets the CSR representative focus 100% of their time on solving the problem, which increases the odds the customer is going complete the contact feeling positive about the experience.
Effort Saving up to 30% where there is no need for CSR to be available all the time for Customer initial queries.
Different way of responses are been designed and tested like following:-
No response — for instance, if someone is idle for sometime.
An invalid response — the wrong message entirely not relevant to the chat.
A valid response with invalid data —  the correct container of query but with an exclusive word.
A valid response — any of a number of variations.
Bot only Chat  95% time savings for CSR
Bot + CSR Chat  5% time savings for CSR
CSR only Chat  0.05% time savings for CSR
Client Name: Farmers Insurance
NPS: A metric to measure customer feedback
GWP: Total premium written and assumed by an insurer before deductions
Combined Ratio: (Incurred Losses + Expenses)/Earned Premium