Agile Test Strategies in Practice: Tales of Large-scale Agile projects 8 May 2015 Geoff Meyer, SW Engineering Director, Dell Inc

Introduction to Key Agile Testing concepts Real-world examples − from multiple, large-scale projects Organizational strategies & considerations Test automation strategies and tooling considerations Test Data Management An In-depth Tools Discussion Focused on how to develop Test Automation IS NOT Session Objectives

Geoff Meyer, SW Director Dell Commerce Services Group Personal Family is paramount; my wife is a Professional Networking queen and my kids make me proud! Lives Austin, TX Lived San Diego, Twin Cities Hobbies Travel, Avid fan of NFL, NHL, EPL, MLB, Golfer, Monday Morning Quarterback Personal Family is paramount; my wife is a Professional Networking queen and my kids make me proud! Lives Austin, TX Lived San Diego, Twin Cities Hobbies Travel, Avid fan of NFL, NHL, EPL, MLB, Golfer, Monday Morning Quarterback Business Specialty Change Agent and champion for Agile transformations Passion Solving problems with great people and enjoying every workday Operating Model Embrace Change and Continuous Learning Roles Director, Architect, Program Manager, Manager, Developer Business Specialty Change Agent and champion for Agile transformations Passion Solving problems with great people and enjoying every workday Operating Model Embrace Change and Continuous Learning Roles Director, Architect, Program Manager, Manager, Developer My Dell Leads Test and DevOps functional areas for Dell.com Mission Implement Continuous Testing, Integration and Delivery practices Goal Transform Dell.com into a SW Development engine characterized by Quality, Predictability and Continuous Delivery My Dell Leads Test and DevOps functional areas for Dell.com Mission Implement Continuous Testing, Integration and Delivery practices Goal Transform Dell.com into a SW Development engine characterized by Quality, Predictability and Continuous Delivery

Agenda Agile Testing Concepts –Organization and Architecture –Agile Testing –Test Automation –Continuous Testing –Large-Scale Agile and SAFE Agile Projects in Action

Agile Roles/Responsibilities 5 Program Product Owner Scrum Master Development Test Small teams Co-located Multi-disciplined Feature-based … Program Manager SW Architect Test Architect DevOps Distributed Projects Scrum of Scrums

Week 1Week 2 Agile Sprint Testing Cycle Manual Regression Automated Regression Automated Regression Automated Regression Sprint Planning Automated Regression Automated Regression Sprint Review Retro spective Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Product Owner Scrum Master Development Test

Automation Pyramid 7 CAUTION: Architecture (and Organization!) can be an enabler or inhibitor “Automate Within the Sprint. Automation on a Scrum project is not optional.” - Mike Cohn

3 Agile Testing Quadrant 8 More Agile Testing: Learning Journeys for the Whole Team. Gregory, Janet; Crispin, Lisa ( ). Feature Team System Team(s) UAT E2E Performance

Automation Tooling 9 IDE UI Tools Scripting Languages Services Unit Frameworks MS-Test

Scaled Agile Framework™ Big Picture © Scaled Agile, Inc and Leffingwell, All rights reserved

Agile Projects in Scale

Project Profiles Application Architecture Presentation Layer Business/Service Layer Data Access Layer Moe Systems Mgmt Application Curley eCommerce UI Application Larry eCommerce Services

Organizing Team Roles/Responsibilities 13 Program Product Owner Scrum Master Development Test

Agile Test Strategy: Moe All Agile Testing managed within the program

Agile Test Strategy: Curley UAT and ‘ility’ testing conducted by external teams UAT Regression tests automated by the program team

Agile Test Strategy: Larry ‘ility’ testing conducted by external teams

Automation Strategy: Moe UI Automation limited to Scenario-based workflows In-Sprint Automation at the Unit and Service Levels

Confidential 18 Functional Validation Performance- Characterization and non-Functional Validation Longevity 11 M2 1.0 SST Baseline M1 Scale Concurrency Simulation Large-ScaleLarge-Scale Large-ScaleLarge-Scale … Sprint 15 ‘ility Testing’; Moe Sprint 21 Key Dependency: 3 rd party HW available 3 rd party user stories completed

Automation Strategy: Curley UI Automation to Scenario- based workflows +UI functional tests Functional Test Automation done In-Sprint at the Unit and UI Levels Services Virtualization to support Continuous Testing

Automation Strategy: Larry Functional Test Automation done in-Sprint at the Unit and Service Levels MS-Test Services Virtualization to support Continuous Testing

Continuous Integration & Automated Testing: Moe 21 Automated Testing Continuous BVT Team #1 Feature Test Exploratory Testing Acceptance Testing Team #2 Feature Test Exploratory Testing Acceptance Testing Team #N Feature Test Exploratory Testing Acceptance Testing CI/Build TFS TeamBuild To Factory

Continuous Integration & Continuous Testing: Curley 22 Continuous Testing Continuous BVT SIT App Test VM Team #1 Feature Test Exploratory Testing Acceptance Testing App Test VM Team #2 Feature Test Exploratory Testing Acceptance Testing App Test VM Team #N Feature Test Exploratory Testing Acceptance Testing … CI/Build TFS TeamBuild To Staging & Production

Continuous Delivery 23 CI/Build BVT Tests BVT Automated Smoke DVT UAT Production Baseline Load Stress Performance E2E Tests UAT regression Staging UAT Func. regression UAT regression E2E SIT Unit test Integration test Dev Test Unit test Integration test Dev Test Unit test Integration test Dev Test Unit test Integration test Dev Test Acceptance Tests Exploratory Tests UI Tests Func. Regression UAT regression App Test Acceptance Tests Exploratory Tests UI Tests Func. Regression UAT regression App Test Acceptance Tests Exploratory Tests UI Tests Func. Regression UAT regression App Test Unit test Integration test Dev Test Static Analysis Auto. Unit Test Gated Check-ins Minutes Accept Auto Depoy Auto Deploy Zero-Touch Deploy Auto Deploy Automated Smoke, DVT, BVT & Regression Suites Continuous Integration–Fast feedback/Green Builds On-demand Environment Provisioning & Deployment Local Control of Environments: Mocks & Data

The Lessons of ‘Large’ There is no cookie-cutter Test Strategy Strive for co-located Teams Establish Culture of Automation Create Cross-team roles Inspect and Adapt

Dell - Restricted - Confidential Recommended Readings 25

Questions?