1. Agile Testing @ Large Geoff Meyer, geoffrey_meyer@dell.com May 2014 Last updated: 25 February, 2014 Courtesy: Gulliver's Travels (2010)

2. Session Objectives Challenges to expect with Agile test practices in large-scale, multi-team projects Brief overview of Industry Approaches Real-world examples − from a large organization perspective Organizational strategies & considerations Test automation considerations An In-depth Tools Discussion Focused on how to develop Test Automation IS NOT

3. Geoff Meyer Dell Inc, 1998 – present – Test Architect › Agile Test & Automation Strategy – Agile Steering Committee Co-Chair – Global Projects › 15 Scrum teams (2 regions) › 11 Scrum teams (4 regions) › 9 scrum teams (2 regions) › 7 Scrum teams (4 regions) NCR Corp. 1984 – 1998 – SW developer, Project Lead, SW Manager B.S. Computer Science, San Diego State University Masters Engineering Management - NTU 3

4. 4 Agenda The Dell Context and Early Agile Testing Challenges In Search of… Solutions for Scale Adaptations for Testing at Scale Automating in an Agile world Lessons of Large Agile @

5. 5 The Context at Dell ESG http://amazngwallpapers.blogspot.com

6. Dell Enterprise Solutions Group Global Design and Development using Agile Scrum 6 Noida Design Center Bangalore Design Center Austin Design Center Silicon Valley Design Center

7. Dell Enterprise Solutions Software Products Server Systems Management Converged Infrastructure Systems Management Private Cloud Systems Management Console Plug-ins (i.e. for SCCM, vCenter…) 7 Common Product Characteristics: Embedded Software Large hardware support test matrix Software is installed in the Data Center Enterprise update cycles ~6 months Products must function even as underlying HW, FW, BIOS, and Drivers are upgraded - SUSTAINING Storage Network Compute

8. The Early Challenges 8 ? http://cleaneatingchelsey.com/

9. Culture Transition Early commitment on Requirements Insufficient Detail in Requirements Development vs. Test A different “School” of Test 1 Fail Early, Inspect & Adapt 9 1 – Scott Barber “Approaches to Software Testing: An Introduction “Approaches to Software Testing: An Introduction “ “Everybody, every team, every platform, every division, every component is there not for individual competitive profit or recognition, but for contribution to the system as a whole” W. Edwards Deming

10. Signs of Trouble Thursday’s Build Committing to more stories than team’s sprint velocity High % of carry-overs Inability to Automate Acceptance Tests within Sprint “Engineering” Stories Large-Scale Cross-team interdependencies block sprint progress Large HW configuration validation consumes test resources No single team had visibility to the entire product Insufficient Product Owner bandwidth http://www.huffingtonpost.com/2012/03/21/truck-stuck-thomas- circle_n_1369545.html?p=truck+and+underpass&vm=r

11. Test Automation Pitfalls Minimal guidance to Test beyond “Go forth and automate” Black Box Test Mentality Automation was interpreted by many as “Automate the UI” 11 Insufficient SW engineering skills across Test teams Architecture(s) not optimized for Automatability

12. 12 In Search of… Solutions For Scale

13. Leverage the Community 13 Books Conferences Professional Groups Consultants

14. Scaled Agile Framework™ Big Picture © 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved

15. Agile Testing Quadrants 15 4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ 4

16. Automation Tooling 16 IDE UI Tools Scripting Languages Services Frameworks

17. ALM Tooling 17 Multi-team, geographically distributed projects create a necessity for ALM Tooling – Requirements Management – User Story Management – Test Case Management – Defect Management – Test Script Repository

18. 18 Agile @ Dell Adaptations

19. Agile Testing @ PG Enterprise Solutions Group How did we go about it? 19 Process/Project Culture/Organization X-Large project (15) 2011 2014 UI Automation “Automate-First” Service-level automation ALM using TFS In-sprint Automation Accelerate Automation Skillset Maximize Utilization Automation Architects Automated Setup & Config 4 Geography project (11) 2012 Defect Prevention Continuous Testing Non-Functional Scale, Longevity 2010 2013 Test into Scrum 2009 Collaborative Seating Rally

20. Agile @ Dell Core Activities Pre-Sprint activities: – Staffing – Training - Project tools and processes – CI/Build environment – Automation Framework and BVT Establish Project-wide ‘Done’ criteria End-to-end, short duration User Stories Test Automation is included User Story acceptance criteria Refresher Workshops for new projects 20

21. Week 1Week 2 2 week Sprint Lifecycle Example Manual Regression Automated Regression Automated Regression Automated Regression Sprint Planning Automated Regression Automated Regression Sprint Review Retro spective Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Product Owner (or Proxy) Scrum Master Development Test Tech Pubs UI

22. 22 In-Sprint: Testing Design/Develop Test cases and Review with team Develop test automation and maintain in TFS Execute test cases (manual and automated) Immediate Feedback (IM, email, in-person) with team of sightings… do not automatically enter defects Verify fixes for defects; re-test if needed Maintain test case status (in MTM) Cross-team coordination including Weekly SoS Confidential

23. Agile @ Dell Roles/Responsibilities 23 Product Owner Product Owner Proxy 1 per Scrum Scrum Master 1 per Scrum Development 4-5 per Scrum Test 3:1 Tech Pubs 1 per 3 scrum team UI 1 per Scrum Scrum teams Small teams Co-located Multi-disciplined Feature-based Distributed Projects Scrum of Scrums Teams formed from functional silos … Release Manager SW Architect Test Architect Automation Architect

24. Cross-team Test roles 24 Test Architect – Defines Test Strategy across all Scrum teams – Establishes non-Functional Test approach – Influences consistent Application Architectures Automation Architect – Leads Analysis and POC activities for new automation needs – Ensures automation consistency for BVT and nightly regression suites – Safeguards against automation tool proliferation

25. EXAMPLE - Project-wide “Done” criteria 25 Code – Junit Tests – SONAR – Code Reviews completed – All User Story tasks completed – Demo and accepted by Product Owner – All known error conditions caught and handled appropriately – Code is Globalized Test – Acceptance Tests Reviewed with scrum team – Acceptance Tests documented in Tool – All planned Acceptance Tests for User Story has been attempted – Acceptance Tests executed against Test system configuration – Defects that are “must fix” before releasing must be fixed prior to story acceptance – Service-level automated tests – Service-level automated Tests are globalized

26. Large-Scale Complexities 26 CharacteristicDescription Large-ScaleProject consists of more than 4 Scrum teams InterdependencyRequirements implemented across multiple Scrum teams Extensive Configuration Matrix Extensive HW or SW configurations Waterfall Intersection One or more components are managed using Waterfall GeographyProject members are Geographically dispersed

27. Development Usability Test Wireframes Architecture Automation FW/Tools Release Planning Pre-sprints HW resources Pre-sprints : Agile with Adaptations 27 Hardening 123N-1N 123 N 123 N …Sprints DefinePlanDevelopLaunch OLP … Software System Test Extended Sprint Test Stability Agile @

28. Extended Sprint Test Extended Hardware Configurations Optimizes HW utilization and cost in support of Scrum teams Owns Test Execution against extended configurations Scrum team: – Establishes Acceptance criteria – Identifies all Test scenarios – Owns Test Execution against reference configuration(s) – Develops in-sprint automation tests 28 Configurations Positive XXXX Negative XXX Boundary XXX Stress XX Scalability XXX Concurrency XXX Globalization XX Candidate for Extended Sprint Test

29. Pre-SST Software System Test “ Alpha Customer” 29 Hardening Workflow Design Test Analysis Test Design Test Development Test Automation Development Test Execution Regression Test Test Execution Regression Test SST ISIS NOT Requirements-basedFunctional testing Customer-usage basedFirst time integration test of workflows Interoperability coverage of HCL in specific customer environments Exhaustive interoperability matrix coverage of the HCL Three phased approachPerformance baseline testing

30. SAFe 1 & 30 Agile @ 1- Courtesy of Dean Leffingwell and Al Goerner of Valtech Scrum teams System team

31. PG Enterprise Validation Agile & Waterfall Converged Model Dell Confidential 31 SST Hardware Product Test EST Final EST N EST 1 Sprint1Sprint2 Sprint N-1 … SprintN Sprint1Sprint2 Sprint N-1 SprintN Sprint1Sprint2 Sprint N-1 SprintN Hardening Stability Sprint Extended Sprints SW Regression Test Solution stack with software focus … SST EST Exit CCB Review Code Freeze A-Rev Final Extended Regression FVFV PT Stability SST A-Rev Candidate SSI … Stability Sprint RC-0 Regression HW-Specific User Stories Complete HW Enablement Features Complete Vendor Deliverables Required 100% PT Attempted All Stories Accepted Feature Complete X-on-A Stories Accepted

32. PG Enterprise Validation Agile @ Dell Geography 32 Co-location of a Scrum team is always preferred Limit geographical distribution of Scrum team to no more than two time-zones Scrum teams within a project can be distributed

33. 33 Automating in an Agile World

34. Opportunities for Automation It’s not just about automating test cases Can also provide efficiencies to: – Infrastructure Automation › Test Preparation, Setup and Configuration – Non-Functional Testing (“ility”) › Longevity, Scale and Performance Characterization – Compatibility Testing (Solution and Device) 34

35. Infrastructure Automation Bare-metal Deployment Setup & Configuration – OS Provisioning and Configuration Test case staging Environment Cleanup/Baseline Virtualization-based test environments 35

36. Test Content Development UI automation on Customer Usage workflows 36 CAUTION: Application Architecture can be an enabler or inhibitor Unit Test Automation In-Sprint, Automated Acceptance Tests Web-services (or CLI) Test Automation for Functional Testing Automated CI (UT, Build, BVT) 2 - Agile Testing: A Practical Guide for Testers and Agile teams 2

37. Application Architecture Matters Does it facilitate testing at the API/Services level? Does the Business/Error Logic reside below the services level? Does the UI architecture support test automation other than record-playback? 37 Large Scale Organizations: Standardize UI architectures across the product portfolio HTML5, Flex, Silverlight… Standardize Service architectures across the product portfolio SOAP, REST, CLI, API

38. Non-Functional Testing Performance Characterization Longevity Stress Scale Concurrency 38 Often times analysis of these areas are simply too cost-prohibitive to be done manually 3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ 3

39. Confidential 39 Functional Validation Performance- Characterization and non-Functional Validation Longevity 11 M2 1.0 SST Baseline M1 Scale Concurrency Simulation Large-ScaleLarge-Scale Large-ScaleLarge-Scale 1213 171819 … Sprint 15 Performance Characterization Timeline 14 16 Sprint 21 Key Dependency: 3 rd party HW available 3 rd party user stories completed

40. The Lessons of ‘Large’ 40 Courtesy: Gulliver's Travels (2010)

41. The Lessons of ‘Large’ 1. Co-located Teams 2. Project-wide ‘Done’ Criteria 3. Culture of Automation 4. System Test Team 5. Test Architect 6. ALM Tooling

42. Questions? 42

43. Resources Agile Manifesto - http://agilemanifesto.org/ Agile Manifesto Articles/Blogs: – http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ - SmartBear http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ – http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ – http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&t t=siteemail&iDyn=2 – Rajini Padmanaban http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&t t=siteemail&iDyn=2 – Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTesthttp://www.utest.com/int-v1/a/beyond-agile-testing – Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael Bolton http://www.satisfice.com/blog/archives/856 Books: – Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell Scaling Software Agility: Best Practices for Large Enterprises – How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo How Google Tests Software – Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory Agile Testing: A Practical Guide for Testers and Agile teams – http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham http://www.dorothygraham.co.uk/automationExperiences/index.html – Unblock! A Guide to the New Continuous Agile – Andy Singleton Presentations: – Approaches to Software Testing: An Introduction – Scott Barber Approaches to Software Testing: An Introduction – Scaled Agile Framework Big Picture – Dean Leffingwell Scaled Agile Framework Big Picture 43

44. Backup 44