Move from Scripted Manual Testing to Scenario-Based Testing Andrea Fox Sr. Test Analyst October 4, 2017

Remembering the co- founder and CEO of Cerner. A life lost too soon. Dedication of talk Remembering the co- founder and CEO of Cerner. A life lost too soon. “The future is not guaranteed, but I like our chances of being the ones who can positively impact the delivery of care.” ~Neal Patterson I wanted to dedicate my talk to Neal Patterson the founder and CEO of Cerner. Neal was an inspiration to many of his associates, but it was his public speaking style that made me believe I too could be a speaker. He was very real and a lot of the times would get lost in thought etc. but it was his realness and his passion that made it worth hearing him.

Summary Identifying the issue Behavior driven tests How we arrived at today’s scenarios What are scenario tests? Tips for writing Test Evidence Pros and Cons of scenario testing Q&A Who here writes/runs tests manually? Are you using scripts?

The issue The problem – the manual, scripted tests were not enough to find defects Only covered a single path Corrective action was to add another test that scripted the failure scenario Spent too much time writing, running, and maintaining (sometimes) Test repository was out of control

BDT – Behavior Driven Testing Had us look at the system from the outside to outline behavior Started to make us think about what is being tested

BDT - Example Given the account has a balance of $2000 When the customer requests a withdrawal of $100 Then the account is debited by $100, and the account balance updates to $1900

Why BDT didn’t necessarily work Written in given, when, then statements Writing scenarios in the description area as well as the steps Spent more time trying to write them then run them and hard to understand Good start – not quite what we needed Behavior driven testing started us on the right track. We were starting to think more about what needs to be tested, but got caught up in the design aspect. We had a hard time writing the tests in the given, when, then statements and the engineers had a hard time wrapping their head around it while testing.

The Transition Originally presented to the group the idea of using exploratory testing Formed the idea of writing steps as scenarios When talking about exploratory I had talked about how we didn’t need the direction the steps were providing but leaving it open for the tester to interpret giving them more freedom. The director didn’t like the idea just having requirements so mentioned scenarios.

What is scenario testing? Definition: 1)an outline of the plot of a dramatic work, giving particulars as to the scenes, characters, situations etc. 2) an imagined or projected sequence of events, especially any of several detailed plans or possibilities In testing it’s the goal or action the user wants to accomplish Scenario testing is the happy medium between scripted testing and exploratory testing.

Best practice tips for writing scenario tests The scenarios tell a story. Can be simple to complex. The test isn’t written to just pass, but to make the tester want to look at all of the angles The scenarios are credible. Can this actually happen to a client? If so it should be included Evaluate whether the test scenarios passed or failed. Are they written in a way that makes the results obvious? Put the time into planning – mind maps, outlines, lists, etc. Kaner, Cem. (2003) An Introduction to Scenario Testing. http://www.kaner.com/pdfs/ScenarioIntroVer4.pdf

Example of Mind Map

Test Evidence Just as important as the scenarios! Evidence should show the story that was tested Make sure to include any thing you deem relevant to proving the scenario passed or failed: screenshots, explanations of what was tested, pass/fail status, etc.

Test evidence example

Pros and Cons Took less time to write Gain solution knowledge faster Allows for multiple paths to be taken during testing Places focus on what is being tested All members of the team (engineers, architect, solution designer, and tester) take part in scenario planning sessions If a scenario fails testing doesn’t have to necessarily stop Can be hard to run without solution knowledge Spend more time planning Not always the best choice for early testing Can expose more design flaws than actual code problems (not necessarily bad)

Any questions?

Thank you for attending! Andrea Fox 2800 Rockcreek Pkwy Kansas City, MO 64117 816-201-4885 andrea.fox@cerner.com