1. Changing Developer Behavior Using Automatic Test Intel Agile and Lean Development Conference - 2014 Chris Gearing 23 rd May 2014 Version 1.0

2. Why Automated Testing Geographically and Organizational Separated Team Needed to Transition from Accepting Defects to Aiming For Perfection Automated test provided a Repeatable Cornerstone to this Change 2 Intel Agile and Lean Development Conference - 2014

3. About Our Team

4. Where we came from Whamcloud –Startup formed in 2010 –Team of ~20 mostly from Oracle ~20 People in ~20 places 3 Continents, 10 time zones Acquired by Intel in 2012 ~50+ people in ~50+ places 4 Continents, more time zones Intel Agile and Lean Development Conference - 2014 4

5. Lustre* File System The world's fastest, most scalable file system Parallel shared POSIX file system –Scalable –High performance –Petabytes of storage –Tens of thousands of clients Complex –350,000+ lines of code –50+ active developers @ 10+ organizations 5 Intel Agile and Lean Development Conference - 2014

6. Lustre* is Open Source Open Source –All that matters is the source –Culturally difficult to tell people how to do things Even those you pay! –Strong culture of independence Code arrival maybe the first contact –Funded projects have plans –But light weight is the way 6 Intel Agile and Lean Development Conference - 2014

7. My Role Asked to come joined Whamcloud and ‘sort testing out’ No testing background No Linux, HPC or Lustre knowledge I had read ‘The Toyota Way’ 7 Intel Agile and Lean Development Conference - 2014

8. How We Worked

9. Everyone had huge commitment Everyone Tested, Reviewed before landing Everyone did it differently Caused a lot of failing tests –People deprioritized the failures –While people ignore the testing you get new failing tests. 9 Intel Agile and Lean Development Conference - 2014

10. How Did We Get Out Of This Situation? 10 Intel Agile and Lean Development Conference - 2014

11. Automated Test What They Wanted – Easy Variation Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option 10 Option 11 Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 Feature 6 Feature 7 Feature 8 Feature 9 Feature 10 Feature 11 Type 1 Type 2 Type 3 Type 4 Type 5 Type 6 Type 7 Type 8 Type 9 Type 10 Type 11 11 Intel Agile and Lean Development Conference - 2014

12. Automated Test What They Got - Repeatability 12 Intel Agile and Lean Development Conference - 2014

13. Automated and Repeatable Test and Landing Process Automated Repeatable Test Tool Driven Distributed Review Traceable Collateral Acceptance 13 Intel Agile and Lean Development Conference - 2014 git push origin HEAD:refs/for/master

14. What Happened

15. At first very variable results Test 1Test 2Test 3Test 4 But the developers knew it was variability in Lustre* not the test system Developers gained trust in the test system 15 Intel Agile and Lean Development Conference - 2014

16. What Happened Focus on Lustre* variability Test 1Test 2Test 3Test 4 Automated Repeatable testing facilitated push for repeatable results But we had trouble getting everything to pass 16 Intel Agile and Lean Development Conference - 2014

17. What Happened Focus on Failing Test Month 1Month 2Month 3Month 4 Repeatable testing and results allowed drive for 100% passing Every test passes or no landing 17 Intel Agile and Lean Development Conference - 2014

18. Landing Test Performance Percentage of landings that passed all tests over first year 18 Intel Agile and Lean Development Conference - 2014

19. What we learnt

20. About ourselves How did previous releases which required 100% pass ever happen? Red changed to Green where the tests were known failures * Many ‘known’ failures! 20 Intel Agile and Lean Development Conference - 2014

21. Make failing hard Resist requests to optimize failures Make The ‘Good’ Path Easy 21 Intel Agile and Lean Development Conference - 2014

22. People do the wrong thing for the right reasons Do we all follow the processes? Not when we think outside factors justify it Pressure causes quiet short-cuts Automated, repeatable and auditable process removes the quiet short cut option 22 Intel Agile and Lean Development Conference - 2014

23. No process creates pressure #1 reason for reduced test? ‘Management’ pressure to land code True or not? Automated repeatable test flushes out the truth 23 Intel Agile and Lean Development Conference - 2014

24. No one wants to be the fall guy Change difficult to bring about when only the ‘good guys’ do it People happy with group changes that they know will happen Automation meant no one got a bad deal Intel Agile and Lean Development Conference - 2014 24

25. Summary Repeatable Test requires Automated Test –Automation Scales The team changed from fault tolerant to a fault intolerant –Consistent Test Helped Provide A Standard People Like Working To A Standard –People Actually Like enforced Team Rules 25 Intel Agile and Lean Development Conference - 2014

26. Legal Notices This presentation is for informational purposes only. INTEL MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. [BunnyPeople, Celeron, Celeron Inside, Centrino, Centrino Inside, Core Inside, i960, Intel, the Intel logo, Intel Atom, Intel Atom Inside, Intel Core, Intel Inside, the Intel Inside logo, Intel NetBurst, Intel NetMerge, Intel NetStructure, Intel SingleDriver, Intel SpeedStep, Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo, Intel StrataFlash, Intel Viiv, Intel vPro, Intel XScale, InTru, the InTru logo, InTru soundmark, Itanium, Itanium Inside, MCS, MMX, Pentium, Pentium Inside, skoool, the skoool logo, Sound Mark, The Journey Inside, vPro Inside, VTune, Xeon, and Xeon Inside] are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Microsoft, Windows, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries. Java and all Java based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. Bluetooth is a trademark owned by its proprietor and used by Intel Corporation under license. Intel Corporation uses the Palm OS® Ready mark under license from Palm, Inc. Copyright © 2014, Intel Corporation. All rights reserved.

28. Backup

29. Scalable Quality Community Tools 29 JIRA Change Request Developer Carries Out Work Developer Tests with Casco - results to Maloo Developer Lands Using Gerrit Jenkins Builds Patched Lustre Autotest Tests Jenkins Build Tests results to Maloo Peer review using Gerrit Patch landed @ Whamcoud – SCRUM process managed with JIRA 29 Intel Agile and Lean Development Conference - 2014

30. Separate Landing From Test 30 Intel Agile and Lean Development Conference - 2014

31. What we learnt Allow extra specific user testing Regular TestSpecial Test Special testing is extra testing + 31 Intel Agile and Lean Development Conference - 2014