1. Agile Test Strategies in Practice: Tales of Large-scale Agile projects 8 May 2015 Geoff Meyer, geoffrey_meyer@dell.com SW Engineering Director, Dell Inc

2. Introduction to Key Agile Testing concepts Real-world examples − from multiple, large-scale projects Organizational strategies & considerations Test automation strategies and tooling considerations Test Data Management An In-depth Tools Discussion Focused on how to develop Test Automation IS NOT Session Objectives

3. Geoff Meyer, SW Director Dell Commerce Services Group 2015 2010 2005 2000 1998 1995 1990 1984 Personal Family is paramount; my wife is a Professional Networking queen and my kids make me proud! Lives Austin, TX Lived San Diego, Twin Cities Hobbies Travel, Avid fan of NFL, NHL, EPL, MLB, Golfer, Monday Morning Quarterback Personal Family is paramount; my wife is a Professional Networking queen and my kids make me proud! Lives Austin, TX Lived San Diego, Twin Cities Hobbies Travel, Avid fan of NFL, NHL, EPL, MLB, Golfer, Monday Morning Quarterback Business Specialty Change Agent and champion for Agile transformations Passion Solving problems with great people and enjoying every workday Operating Model Embrace Change and Continuous Learning Roles Director, Architect, Program Manager, Manager, Developer Business Specialty Change Agent and champion for Agile transformations Passion Solving problems with great people and enjoying every workday Operating Model Embrace Change and Continuous Learning Roles Director, Architect, Program Manager, Manager, Developer My Role @ Dell Leads Test and DevOps functional areas for Dell.com Mission Implement Continuous Testing, Integration and Delivery practices Goal Transform Dell.com into a SW Development engine characterized by Quality, Predictability and Continuous Delivery My Role @ Dell Leads Test and DevOps functional areas for Dell.com Mission Implement Continuous Testing, Integration and Delivery practices Goal Transform Dell.com into a SW Development engine characterized by Quality, Predictability and Continuous Delivery

4. Agenda Agile Testing Concepts –Organization and Architecture –Agile Testing –Test Automation –Continuous Testing –Large-Scale Agile and SAFE Agile Projects in Action

5. Agile Roles/Responsibilities 5 Program Product Owner Scrum Master Development Test Small teams Co-located Multi-disciplined Feature-based … Program Manager SW Architect Test Architect DevOps Distributed Projects Scrum of Scrums

6. Week 1Week 2 Agile Sprint Testing Cycle Manual Regression Automated Regression Automated Regression Automated Regression Sprint Planning Automated Regression Automated Regression Sprint Review Retro spective Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Product Owner Scrum Master Development Test

7. Automation Pyramid 7 CAUTION: Architecture (and Organization!) can be an enabler or inhibitor “Automate Within the Sprint. Automation on a Scrum project is not optional.” - Mike Cohn

8. 3 Agile Testing Quadrant 8 More Agile Testing: Learning Journeys for the Whole Team. Gregory, Janet; Crispin, Lisa (2014-09-30). Feature Team System Team(s) UAT E2E Performance

9. Automation Tooling 9 IDE UI Tools Scripting Languages Services Unit Frameworks MS-Test

10. Scaled Agile Framework™ Big Picture © 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved

11. Agile Projects in Action @ Scale

12. Project Profiles Application Architecture Presentation Layer Business/Service Layer Data Access Layer Moe Systems Mgmt Application Curley eCommerce UI Application Larry eCommerce Services

13. Organizing Team Roles/Responsibilities 13 Program Product Owner Scrum Master Development Test

14. Agile Test Strategy: Moe All Agile Testing managed within the program

15. Agile Test Strategy: Curley UAT and ‘ility’ testing conducted by external teams UAT Regression tests automated by the program team

16. Agile Test Strategy: Larry ‘ility’ testing conducted by external teams

17. Automation Strategy: Moe UI Automation limited to Scenario-based workflows In-Sprint Automation at the Unit and Service Levels

18. Confidential 18 Functional Validation Performance- Characterization and non-Functional Validation Longevity 11 M2 1.0 SST Baseline M1 Scale Concurrency Simulation Large-ScaleLarge-Scale Large-ScaleLarge-Scale 1213 171819 … Sprint 15 ‘ility Testing’; Moe 14 16 Sprint 21 Key Dependency: 3 rd party HW available 3 rd party user stories completed

19. Automation Strategy: Curley UI Automation to Scenario- based workflows +UI functional tests Functional Test Automation done In-Sprint at the Unit and UI Levels Services Virtualization to support Continuous Testing

20. Automation Strategy: Larry Functional Test Automation done in-Sprint at the Unit and Service Levels MS-Test Services Virtualization to support Continuous Testing

21. Continuous Integration & Automated Testing: Moe 21 Automated Testing Continuous BVT Team #1 Feature Test Exploratory Testing Acceptance Testing Team #2 Feature Test Exploratory Testing Acceptance Testing Team #N Feature Test Exploratory Testing Acceptance Testing CI/Build TFS TeamBuild To Factory

22. Continuous Integration & Continuous Testing: Curley 22 Continuous Testing Continuous BVT SIT App Test VM Team #1 Feature Test Exploratory Testing Acceptance Testing App Test VM Team #2 Feature Test Exploratory Testing Acceptance Testing App Test VM Team #N Feature Test Exploratory Testing Acceptance Testing … CI/Build TFS TeamBuild To Staging & Production

23. Continuous Delivery 23 CI/Build BVT Tests BVT Automated Smoke DVT UAT Production Baseline Load Stress Performance E2E Tests UAT regression Staging UAT Func. regression UAT regression E2E SIT Unit test Integration test Dev Test Unit test Integration test Dev Test Unit test Integration test Dev Test Unit test Integration test Dev Test Acceptance Tests Exploratory Tests UI Tests Func. Regression UAT regression App Test Acceptance Tests Exploratory Tests UI Tests Func. Regression UAT regression App Test Acceptance Tests Exploratory Tests UI Tests Func. Regression UAT regression App Test Unit test Integration test Dev Test Static Analysis Auto. Unit Test Gated Check-ins Minutes Accept Auto Depoy Auto Deploy Zero-Touch Deploy Auto Deploy Automated Smoke, DVT, BVT & Regression Suites Continuous Integration–Fast feedback/Green Builds On-demand Environment Provisioning & Deployment Local Control of Environments: Mocks & Data

24. The Lessons of ‘Large’ There is no cookie-cutter Test Strategy Strive for co-located Teams Establish Culture of Automation Create Cross-team roles Inspect and Adapt

25. Dell - Restricted - Confidential Recommended Readings 25

26. Questions?