Surveying – the New Testing? (introducing Paul Gerrard gerrardconsulting.com Slide 1Intelligent Definition and Assurance

Background To modernise our approaches, we need some new ideas Based on the New Model for Testing, I propose a new approach using a Surveying metaphor This metaphor enables us to think differently on how we use tools to support knowledge acquisition – Requires new collaborative tools to capture ideas and findings – Tools prompts testers to ask questions, to capture thinking Your pair in the exploration and testing of systems might soon be a robot In this vision, automated tools get a new role – supportive of tester thinking, but not replacing it. Intelligent Definition and AssuranceSlide 2

Potted history DevOps is of interest and ChatOps too So I build a little bot to react to messages in a Slack chat is a Python program – It listens to a conversation that – Provides responses directly to the chat room – Uses some web services to manage status of activities in a simulated Deployment Pipeline – It’s a simulator, not a product Works OK – could be better, but it got me thinking more broadly about bots. Slide 3Intelligent Definition and Assurance

Bots are in the news All the talk about bots focuses on AI and Deep Learning Bots are increasingly important But I'm interested (for now) in bots that do simple things on behalf of individuals or a team Intelligent Definition and AssuranceSlide 4

Imagine a tester like a doctor in an autopsy Doctor dictates their findings to be transcribed Tester narrates their activity, captured directly I've been experimenting with speech recognition as a front end to a bot engine that is configurable to do (in principle) any task What? Any task? Well, any task that can be off loaded to a web service or message service. Slide 5Intelligent Definition and Assurance

The vision thing I want a bot that will support what is commonly called ‘exploratory testing’ You know my view (model) of testing already Can we use the explore v test paradigm in a bot that allows you to: – Explore, take notes and model – Record ideas, risks, tests, outcomes and bugs – Generate reports and documentation as a consequence, rather than requiring you write them But... Intelligent Definition and AssuranceSlide 6

The target user is… Not a freestyle exploratory tester working in a start-up or small product company Testers working in: – Regulated, high-integrity, safety critical – High documentation, high accountability – Environments where testers are constrained by their processes – Testers struggling to apply ‘trad’ methods in an Agile, Digital, DevOps environment Why can’t we dictate a detailed test plan for a bot to document, analyse, visualise? Intelligent Definition and AssuranceSlide 7

ThinkBot (provisional name) Five components to ThinkBot: – Robot Schema manager – Web2py web app; manages the models (schemas) that controls a robot – The schema for the bot itself – Robot Engine – a Python Module that uses the Robot Schema to provide an interactive, communicative bot – Robot Clients (Python Scripts) Command line Interface – uses text to control the bot Speech Recognition interface – hands-free control of the bot – Robot App/Website – different web apps for different Schemas The Schema Manager and engine are generic – other Schemas and Apps can be developed – 1st was a "State Model Designer" – for testing the concept only – 2nd called "System Surveyor" – Exploratory Testing bot support. Intelligent Definition and AssuranceSlide 8

Robot Harness The Robot Engine is a Python Module which can be imported to any client The Robot Test Harness is a special client that uses the Python unittest module to run tests Creating the Schema Manager, Surveyor website, two clients and Surveyor bot model simultaneously: There are five two moving parts TDD-like approach is essential to keep control Most activity on the Surveyor model and Surveyor website. Intelligent Definition and AssuranceSlide 9

Schematic Intelligent Definition and AssuranceSlide 10 Schema Server Schema Manager (web site) Schema Repository (web service) Robot Client Robot Engine App Server Actions (web services) Application Web Site Speech Recognition Interface Command Line Interface Administer Schemas through the Web Download Schema Perform Robot Actions through services Application Reporting, Monitoring, Control, Management Many App Servers – one (or many) per Schema robotschemas surveyor

Robot Schemas Note that the Robot Schemas design could be use for ANY kind of bot. Surveyor is just the first.

Schema – an enhanced state model States – Represents the current sate or mode of the Bot Transitions – Initial State – Stimulus – to cause the transition – these are the validated commands for the app – Output – current state, or the response from the application server – Inputs, Validations, Actions, Variables (next slide) – Final state Intelligent Definition and AssuranceSlide 12

Transitions Navigation between states Data capture and validation Actions Internal Webservices Other e.g. MQTT, RPC… Entry and exit criteria Maintaining state and data values in context. Slide 13

Bot Schema – State Model+ Slide 14Intelligent Definition and Assurance

Surveyor Schema close-up Intelligent Definition and AssuranceSlide 15 Demo Robot Schema

System Surveyor An exploratory testing bot

Simplified Schema Intelligent Definition and AssuranceSlide 17 Session Notes Observation Navigation Modelling Home Always start here – ready to commence the session In session – everything is done inside a session Places, features, forms and fields comprise the model Notes and observations are related to places, features, forms or fields Notes can be ideas, concerns, risks, questions etc.

New Model Testing 29 page paper: Slide 18Intelligent Definition and Assurance

Surveyor supports Exploration and Testing Testers use the web interface for charters, planning, reporting and client for session work Charters (plans) define the scope of sessions Sessions are unrestricted – typically minutes, but there is no limits – Can explore, model or test, but in fact, you can do all concurrently Sessions are revised and charters reviewed Reporting at any point and continuously Completion on the basis of consensus.

A continuous, incremental process Intelligent Definition and AssuranceSlide 20 Surveyor Web Site Surveyor Client Scoping exploration sessions Charters created, refined and re-used over time Reporting throughout Interactive Testing Sessions Clarifications Continuous Integration Incremental modelling, refinement over time Changes Refinements Continuous Integration isn’t part of the client, but could be

One system model created by teams in chartered sessions Intelligent Definition and AssuranceSlide 21

Demo See over

Demo agenda Robot Schema Create a Charter Command line Client – some useful commands – Charter, locations, new, note, observation, variables Surveyor – reporting and navigation, sessions Speech client demo Surveyor - reporting

Test Process Potentially, Surveyor could be used to create test plans – Initially based on documentation – Evolving plan based on chartered sessions Testing in the small – chartered testing Testing in the large – Surveyor has the building blocks for end to end tests Could Surveyor generate the documentation required in regulated industries? Intelligent Definition and AssuranceSlide 24

Challenges, opportunities Is it feasible to generate a high or low level test plan from Surveyor? How will the testing feature work? Integrations – Chatbot – Slack etc. – Internal IM – Stories – Test automation a la BDD – both ways Visualisations – Coverage – Reporting – Impact analysis Other benefits – Instant replays – Heat map, floor plan – Tester paths – Real time notifications – Real time test(er) management Intelligent Definition and AssuranceSlide 25

Surveying – the New Testing? (introducing Paul Gerrard gerrardconsulting.com Slide 26Intelligent Definition and Assurance