1. Digital Transformation, Testing and Automation @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com Intelligent Definition and AssuranceSlide 1

2. Helping clients transform their testing through INNOVATION, COACHING and LEADERSHIP Our CLIENTS – Want to be agile rather than follow Agile dogma – Have a pragmatic approach and are focused on delivery – Want a solution that fits, not a badly fitting suit. Intelligent Definition and AssuranceSlide 2

3. "Automation is the future!" Heard that before? What exactly is possible and impossible with automation, right here, right now? Where are the Digital, DevOps and Continuous Delivery crazes leading us? Where do testers fit? How do testers work in high-paced, automation- dominated environments? Let's look into the near future and discuss how we survive or thrive. Intelligent Definition and AssuranceSlide 3

4. Agenda PAC Report on Digital Testing What is Digital Anyway? What Goes Wrong with Automation? A Modelling Problem, not a Tools Problem Models of Testing and Automation Future Testing The Tools Knowledge Base Tomorrow's Challenges. Intelligent Definition and AssuranceSlide 4

5. PAC Report: Digital Testing: Strategies, Challenges & Measuring Success http://www.testplant.com/pac-report Intelligent Definition and AssuranceSlide 5

6. Digital is real, important, but only beginning Intelligent Definition and AssuranceSlide 6

7. Testing is moving closer to the business Intelligent Definition and AssuranceSlide 7

8. Automation levels are still too low Intelligent Definition and AssuranceSlide 8

9. What is Digital anyway? Digital is the buzz-phrase of the moment – Social, Mobile, Analytics, Cloud (SMAC) – Consumerization of IT – Buzzword bingo* Agile laggards (gov, large companies) playing catch-up S/W development at the pace of marketing. * https://zoomph.com/blog/top-20-digital-marketing-buzzwords/ Intelligent Definition and AssuranceSlide 9

10. Why is ‘IT moving closer to business’? Marketers in charge? Customer Engagement (CE) focus: – Real-Time benefits *during* CE – Social media is a tool, not a platform – CE simplicity is critical – Data used proactively to manage CE – Leverage and embrace customer feedback – Anticipate and act on customer needs Much more aggressive uses of IT More 'business-focused' IT Product Owners. Intelligent Definition and AssuranceSlide 10

11. Frequent/regular s/w delivery is critical Mobile users expect apps to change almost daily – New features, offers, opportunities all the time Users use the best apps to get what they want – Not necessarily the best supplier Businesses are in an APPS RACE Speed of delivery is... – Partly about pro-action – Partly about survival. Intelligent Definition and AssuranceSlide 11

12. Automation (not just test) is critical Business wants IT responsiveness (true agility) – Not necessarily 100s of releases/day – Rapid, regular turnaround from ideas to software delivery With continuous integration/deployment, DevOps, developers can now promise Continuous Delivery Testers need to provide Continuous Assurance "Automation through the (shortened) lifecycle" But test automation success is still hard to achieve. Intelligent Definition and AssuranceSlide 12

13. A recent survey ? - 1997 Intelligent Definition and AssuranceSlide 13

14. What Goes Wrong with Test Execution Automation?

15. Tools are sensitive instruments Intelligent Definition and Assurance You don't need tools and tools won't help, if your software is unstable. Slide 15

16. Test automation ambition Intelligent Definition and AssuranceSlide 16 We'll have to make a remarkable improvement to make this happen in one year

17. The anti-regression goal What are you trying to do with automation? Anti-regression is the primary goal – Detect unwanted change – Insurance that developers can make changes safely How do you achieve this? – The automation model must align with the developer model – You have to collaborate early to achieve this. Intelligent Definition and AssuranceSlide 17

18. Testing and automation – a modelling problem not a tools problem Need to shift testing thinking to the left. We model to improve requirements, to eliminate mistakes in code and automation.

19. Brian Marick's Original Agile Testing Quadrant Intelligent Definition and AssuranceSlide 19

20. Crispin & Gregory (Agile Testing version) Intelligent Definition and AssuranceSlide 20

21. Intelligent Definition and AssuranceSlide 21

22. Intelligent Definition and AssuranceSlide 22

24. Intelligent Definition and AssuranceSlide 24

25. Are these models useful and still relevant to you?

26. At least three dimensions to automated testing 1.Requirements certainty 2.Business or Technology facing/supporting 3.Architectural/technical testability Of course there are other considerations, but I want to cover number 1 for a moment. Intelligent Definition and AssuranceSlide 26

27. ALL Testing is Exploratory We explore sources of knowledge...... to build test models...... that inform our testing.

28. Judgement, exploring and testing Testing (the system) Our model(s) are adequate Our model(s) are not adequate Exploring (sources) Judgement Creates test models Uses test models We explore sources of knowledge to build test models that inform our testing We don't yet know what the system should do. We can't test yet (learning what it should do) We believe that we know what the system should do. We can (automate) test. (learning what it does do)

29. New Model Testing 29 page paper: http://dev.sp.qa/download/newModelhttp://dev.sp.qa/download/newModel Intelligent Definition and AssuranceSlide 29 Bach/Bolton testing v checking doesn't help. They don't separate exploration from testing...

30. New Model Testing Intelligent Definition and Assurance The Left The Right Slide 30

31. Left and Right Automation won’t work if the thinking to the left is not in place Thinking on the left contributes to the definition of features and how you test them If you aren’t involved on the left, you aren’t in a good position to test (with or without tools) Models on the left are the blueprints for the tests & automation on the right Your opportunity to flush out muddled, loose, incomplete, flaky requirements is on the left Applying automation to muddled, loose, incomplete, flaky features is doomed before you start Intelligent Definition and Assurance The Left The Right Slide 31

32. Shift-Left (thinking, not people) 1.If you are not contributing on the left, you are doomed to be a victim of muddled, incomplete thinking 2.Challenge requirements thru example and improve them 3.Share your models! – Usage models: (test through the UI) must be meaningful to users – Service models: (services/APIs) meaningful to developers and align with business rules 4.At the first sign of testing/automation ‘difficulty’ – go to 1. Intelligent Definition and AssuranceSlide 32

33. Today's Challenge Can tools ever find functional bugs in untested software? Aren't they limited to regression testing? Are they limited because we can't create test models usable by tools? Intelligent Definition and AssuranceSlide 33

34. Future Testing

35. Smart City World Congress Slide 35Intelligent Definition and Assurance

36. Slide 36Intelligent Definition and Assurance

37. Test this? Slide 37Intelligent Definition and Assurance

38. IoT Seven layer architecture Not every system has seven separate layers. Layers are often combined into one object/system Internet connected Data-Centre Based Not Internet connected Local Network Local, metro network internet connected Slide 38 The problem will be to test connected mobile devices – at scale, in the lab and in the field.

39. Tool Support Automated support will be essential and probably dominate testing Most components tested through APIs, web or other services New protocols emerging; custom drivers may be required The challenge: "How do we design the thousands or millions of tests to feed the tools?" Intelligent Definition and AssuranceSlide 39

40. Test Analytics DevOps processes and production analytics monitor system vital signs (like sensors) – These analytics are essentially the same Production systems are platforms for experimentation – The code that serves marketers can also serve developers and testers Analytics code is your sensor network DevOps processes are 'things', too. Intelligent Definition and AssuranceSlide 40

41. Modelling, Testing and Test Data Tools to model the IoE are only just emerging We'll need to use Big Data test techniques 'Pattern-based test design' – Generate tests from patterns to simulate scenarios – 'Heuristic oracles' will derive from conventions or common-sense – Personas as patterns of user or system behaviour Developers will need to make systems testable through High Volume Automated Testing. Intelligent Definition and AssuranceSlide 41

42. Manual v Automated Testing? (wrong question) Automation will not make IoT testing easy; it will make testing possible More and more, DevOps processes are a source of data for analysis just like production (SMAC) (Until the tools/platforms become available) you may need to: – Write your own utilities, download, configure and adapt open source tools or create automated tests – Have informed conversations with developers Should testers learn to code? Probably. Intelligent Definition and AssuranceSlide 42

43. The Tools Knowledge Base https://tkbase.com

44. How many tools do you use? NewRelic - Application monitoring - gives us the eyes on our app and how it's being used / performing PaperTrail - Log file collector - brings in log files from various servers to one single place - great for systems running across multiple servers OpsView - Monitoring and alerting tool which we use to bring together monitoring from various systems Nagios - Used for monitoring and alerting PagerDuty - Used to alert (SMS and Email and Phone) when a service craps out Elastic Search, Log Stash and Kibana - Data analysis and monitoring and trending - super powerful for analyzing what our product is actually doing Chef - Auto build and deploy technology to allow us to rapidly build and destroy environments (with Chef Kitchen and Knife) Vagrant - Create Virtual Environments Real Time Board - Virtual Whiteboard - amazingly useful Pivotal Tracker - Agile tracking tool Fiddler - Proxy web tool Firebug - Proxy web tool Zed Attack Proxy - Security testing tool Burpsuite - Security Testing Tool HipChat - Real Time IM communication tool Slack - Real Time IM Communication tool Jira and Confluence - bug tracking and wiki CloudFormations - Creates templates for Amazon instances "No doubt we have some more hiding away but that's a pretty good list." 23 tools Intelligent Definition and AssuranceSlide 44

45. How many tools are there anyway? I'm researching tools for tkbase.com – 2336 of which 685 are programming languages – 1651 tools for DevOps, SDET & Testers Tool types and features – https://tkbase.com/tools https://tkbase.com/tools My guess is there are at least 2000. Intelligent Definition and AssuranceSlide 45

46. 284 bloggers, 15,022 blogposts indexed and searchable (35-40 new posts daily) 2,336 tools, indexed and searchable Work in progress: tools tagged with 294 features

47. Challenges for the next 5-10 years Where are the large-scale simulation tools going to come from? How do we acquire a coherent, integrated tool-kit with 15-30 tools? How do we integrate tools at all levels into a fast-moving development process? What skills do testers need? Developers? Where does the manual testing fit? Intelligent Definition and AssuranceSlide 47

48. Thank-You! @paul_gerrard Paul Gerrard email: paul@gerrardconsulting.compaul@gerrardconsulting.com Twitter: @paul_gerrard LinkedIn: http://linkedin.com/in/paulgerrard Intelligent Definition and AssuranceSlide 48