1. Automation Culture: Essential to Agile Success Agile Austin Geoff Meyer, geoffrey_meyer@dell.com October 2013 Last updated: September 29, 2013

2. Session Objectives Challenges to expect when adopting test automation during your Agile transition Practical solutions to what & when to automate Real-world examples − from a large organization perspective Challenges automating in a HW-dependent environment For organizational leaders driving test strategy A Tools Discussion Focused on how to develop Test Automation IS NOT

3. Introductions 3

4. Geoff Meyer Dell Inc, 1998 – present – Test Architect › Agile Test & Automation Strategy – Agile Steering Committee Co-Chair – Global Projects › 15 Scrum teams (2 regions) › 11 Scrum teams (4 regions) › 9 scrum teams (2 regions) › 7 Scrum teams (4 regions) NCR Corp. 1984 – 1998 – SW developer, Project Lead, SW Manager B.S. Computer Science, San Diego State University Masters Engineering Management - NTU 4

5. 5 Agenda Why Automation is essential to Agile The Dell Context and Agile @ Dell Challenges and Common Pitfalls The Automation Landscape Foundations of an Automation Culture Care and Feeding of the Automation Culture

6. The Need, The Challenge, The Pitfalls 6 Ø

7. Why is Automation So Important in Agile? Near-term – Ensures that you don’t break what you just built – Provides safety net for developers & rapid feedback to new changes – Continuous Integration and use of Build Verification Test (BVT) Long-term – Maximizes velocity of Scrum team (increased value creation) – Creates capacity for Exploratory and ad-hoc Testing – Enables activities that can’t be done cost-effectively by humans And if you don’t… Quality is at risk from an unmanageable regression suite

8. Differences with Automation in Waterfall? In Waterfall… Automated tests are derived from the backlog of completed features – In Agile, Automation can be incorporated in the requirement Testing and automation is performed after Development is “complete” Focus is on first-time discovery of defects and optimizing your test coverage vs. Agile… where automation development is ongoing and provides immediate feedback throughout development 8

9. 9 The Context @ Dell http://amazngwallpapers.blogspot.com “Apply Test Automation in the context of your organizational automation needs” --- Bob Galen, Agile Coach

10. Dell Enterprise Solutions Group Global Design and Development using Agile Scrum 10 Noida Design Center Bangalore Design Center Austin Design Center Silicon Valley Design Center

11. Dell Enterprise Solutions Software Products Server Systems Management Converged Infrastructure Systems Management Private Cloud Systems Management Console Plug-ins (i.e. for SCCM, vCenter…) 11 Common Product Characteristics: Large hardware support test matrix Software is installed in the Data Center Enterprise update cycles ~6 months Products must function even as underlying HW, FW, BIOS, and Drivers are upgraded - SUSTAINING

12. Agile @ Dell Roles/Responsibilities 12 Product Owner Product Owner Proxy ~1 per Scrum Scrum Master 1 per Scrum Development 4-5 per Scrum Test 3:1 Tech Pubs 1 per 3 scrum teams UI 1 per Scrum Scrum teams Small teams Co-located Distributed Projects Teams formed from functional silos

13. Agile @ Dell with Adaptations 13 Hardening 123N-1N 123 N 123 N … Release Exit Feature Complete Code Freeze Sprints Release Plan DefinePlanDevelopLaunch OLP … Software System Test Extended Sprint Test Stability

14. Pitfalls encountered at Development didn’t historically automate unit tests Build teams were staffed with non-Build practitioners Minimal guidance to Test beyond “Go forth and automate” 14 Black Box Test Mentality Architecture(s) not optimized for Automatability Automation was interpreted by many as “Automate the UI” Insufficient SW engineering background across Test teams

15. Which Resulted In… Over-emphasis on UI automation Test Automation not keeping up within the sprint Feature Devotion 1 Multiple automation tools & licenses Test scripts not designed for re-use 15 1 – “A nasty condition where people start valuing ticking off features more than tracking the real outcome of the project.”--- Martin Fowler

16. 16 The Automation Landscape http://amazngwallpapers.blogspot.com

17. Opportunities for Automation It’s not just about automating test cases Can also provide efficiencies to: – Test Preparation, Setup and Configuration – Non-Functional Testing (“ility”) › Longevity, Scale and Performance Characterization – Compatibility Testing (Solution and Device) 17

18. Test Preparation Bare-metal Deployment Setup & Configuration – OS Provisioning and Configuration Test case staging Environment Cleanup/Baseline Virtualization-based test environments 18

19. Test Content Development Whole team ownership of QA & Automation UI automation on Customer Usage workflows 19 CAUTION: Application Architecture can be an enabler or inhibitor Unit Test Automation In-Sprint, Automated Acceptance Tests Web-services (or CLI) Test Automation for Functional Testing Automated CI (UT, Build, BVT) 2 - Agile Testing: A Practical Guide for Testers and Agile teams 2

20. Application Architecture Matters Does it facilitate testing at the API/Services level? Does the Business/Error Logic reside below the services level? Does the UI architecture support test automation other than record-playback? For Large organizations: – Standardize UI architectures across the product portfolio › HTML5, Flex, Silverlight… – Standardize Service architectures across the product portfolio › SOAP, REST, CLI, API 20

21. Prioritize and Identify what NOT to Automate What NOT to Automate Everything GUI – Except for High-value customer usage flows after the UI is stable Tests that would only find low severity bugs – Where the product is unchanging 21 Prioritize High ROI 1. BVT Candidate Acceptance Test – Test of core functionality that executes in a short duration 2. Core Functionality to be run in Nightly Regression 3. Sustaining Test Candidate Test – Functional test that verifies the application can withstand subsystem changes

22. Non-Functional Testing Performance Characterization Longevity Stress Scale Concurrency 22 Often times analysis of these areas are simply too cost-prohibitive to be done manually 3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ 3

23. 23 Foundations of an Automation Culture Images.yahoo.com

24. Keys To A Culture Transition Development vs. Test A different “School” of Test 4 Evolve from Functional Responsibility … to Whole Product Ownership 24 4 – Scott Barber “Approaches to Software Testing: An Introduction “Approaches to Software Testing: An Introduction

25. Establishing an Automation Culture Identify Near and long-term automation focus areas Inventory the culture and skillset of organization Establish – Tooling and Infrastructure Standardization – Your ‘community’ Develop – Workforce transition plan Organize – Embedded vs. Specialized Automation team – Re-align project staffing Operationalize 25

26. Identify Focus Areas for Automation 26 4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ 4

27. Establish Tooling and Infrastructure – Tooling Standards and Frameworks – Team to develop/manage home-grown tools – Lead tool evaluations to meet new architectures & technologies Community: Internal and External 27 Multiple automation Tools/FW’s may be needed to automate at layers of testing: Unit level Middle tier UI-driven

28. Develop & Organize Adjust Hiring Practices Re-align Project staffing Embedded Automation vs. Specialized Team Training – New-hire – Ongoing Confidential 28

29. Operationalize Whole team commitment in words and actions Include Test Automation in Acceptance Criteria – Unit – Acceptance Continuous Integration and BVT Establish Metrics, Governance & Rewards 29

30. Automation @ PG Enterprise Solutions Group How did we go about it? 30 Process/Project Culture/Organization In-Sprint Automation Established ALT 2011 2014 UI Automation “Automate-First” Service-level automation (CLI & Web-services) IVT Networking & Servers Automated BVT IVT Device Accelerate Automation Skillset Maximize Utilization Automation Architects Setup & Config Internal Training Programs 2012 TDD & Automated UT Non-Functional Scale, Longevity 2010 Automation Day 2013 Automation Content Libraries

31. Care and Feeding of the Automation Culture 31

32. Maintaining the Automation Culture Operationalize Automation – Include Test Automation in the Acceptance Criteria Encourage Community Practice and Participation – External - Industry User Groups & Conferences – Internal - Brown-bag sessions, Showcases, Mini-conference For Large organizations: – Overcome skillset deficiencies by adjusting staffing strategy – Automation Leadership Team, Automation Architect(s) – Continually monitor the alignment of Architecture, Dev and Test – Develop reward systems 32

33. Recognizing a Successful Automation Culture Automation is a shared responsibility Automatability is a key architectural consideration Continuous Integration & BVT Teams are staffed to include an automation skillset Automation is operationalized

34. Resources Agile Manifesto - http://agilemanifesto.org/ Agile Manifesto Articles/Blogs: – http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ - SmartBear http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ – http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ – http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=177 93&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=177 93&tth=DYN&tt=siteemail&iDyn=2 – Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTesthttp://www.utest.com/int-v1/a/beyond-agile-testing – Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael Bolton http://www.satisfice.com/blog/archives/856 Books: – Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell Scaling Software Agility: Best Practices for Large Enterprises – How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo How Google Tests Software – Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory Agile Testing: A Practical Guide for Testers and Agile teams – http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham http://www.dorothygraham.co.uk/automationExperiences/index.html Presentations: – Approaches to Software Testing: An Introduction – Scott Barber Approaches to Software Testing: An Introduction – Agile Testing: Challenges Beyond the Easy Contexts – Bob Galen 34

35. Questions? 35