1. Agile Testing isn’t Risking IT! Wednesday 7 November 2012 Eurostar 2012

2. 2© All rights reserved Agile Testing isn’t Risking IT! Introduction What is it all about? Which problems did I face? How to do it right? Recap - Q & A 1 2 3 4 5 6

3. 3© All rights reserved Agile Testing isn’t Risking IT! Introduction What is it all about? Which problems did I face? How to do it right? Recap - Q & A 1 2 3 4 5 6

4. © All rights reserved Agile in a nutshell 4 Product Backlog Sprint Backlog Sprint 2-4 wk Scrum 24 u Product Owner Team Scrum Master Demo Planning Meeting Retrospective

5. © All rights reserved “A process or method to analyse the risks to the product, with the intention to utilize the testing effort in the most valuable way.” Product Risk Analysis in a nutshell Classification: HighClassification: Medium Classification: LowProduct RiskRequirement “The likelihood that a product fails in relation to the conceivable damage when a product fails in production.” 5

6. 6© All rights reserved Risk & Requirement Based Testing in a nutshell Risk without requirement: Add requirement (a fault found early) Remove Risk (no worries about nothing) Requirement without Risk Add product risk (improves coverage of test) Remove Requirement (only make what is needed) Requirements Product Risks

7. 7© All rights reserved Agile Testing isn’t Risking IT! Introduction What is it all about? Which problems did I face? How to do it right? Recap - Q & A 1 2 3 4 5 6

8. © All rights reserved Risk analysis were not translated to the Backlog which makes it difficult to utilise a risk based approach. Merging or splitting features invalidate the PRA. 8

9. © All rights reserved Risk analysis were not updated every iteration, which could mean that they will be out dated within a sprint. PRA isn’t updated 9

10. © All rights reserved Risk analysis were only done from the requirements which meant that some product risks were missed. PRA are done with tunnel vision Specify Test Cases Specify Test Cases Determine Risk Determine Risk Order Requirements Order Requirements Analyse Requirements Analyse Requirements Analyse Risk Order Requirements Order Requirements Analyse Requirements Analyse Requirements Specify Test Cases Specify Test Cases 10

11. © All rights reserved Risk analysis were done, but the outcome wasn’t taken into account in the planning meetings. Risk class isn’t used in planning meeting Team Velocity = ± 14 Story Points High = 3 n Medium = 1 n Low = ½ n 13

12. © All rights reserved Risk analysis were done, but the risks where prioritised using MoSCoW, which doesn’t automatically fit in a Agile environment. Testing is prioritized based on product risks Sprint 1 Sprint 2Sprint 3 12

13. 13© All rights reserved Agile Testing isn’t Risking IT! Introduction What is it all about? Which problems did I face? How to do it right? Recap - Q & A 1 2 3 4 5 6

14. © All rights reserved Gather Risk Items Gather & Order Requirements Gather Product Risks Plot Risk Items Map Requirements & Product Risks Calculate Risk Items Determine Impact & Likelihood Factors Score & Calculate Risk Items/Factors Strategy to tactic Risk class tactic Sprint tactic Gather Risk Items 14

15. © All rights reserved Gather & Order Requirements Rank # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 REQ 04 REQ 04 REQ 02 REQ 02 REQ 07 REQ 07 REQ 03 REQ 03 REQ 01 REQ 01 REQ 05 REQ 05 REQ 06 REQ 06 REQ 08 REQ 08 REQ 09 REQ 09 REQ 10 REQ 10 REQ 14 REQ 14 REQ 11 REQ 11 REQ 12 REQ 12 REQ 13 REQ 13 REQ 15 REQ 15 17

16. © All rights reserved Gather Product Risks PR 02 PR 05 PR 01 PR 04 PR 03 PR 08 PR 07 PR 06 PR 09 16

17. © All rights reserved Gather Risk Items Gather & Order Requirements Gather Product Risks Plot Risk Items Map Requirements & Product Risks Calculate Risk Items Determine Impact & Likelihood Factors Score & Calculate Risk Items/Factors Strategy to tactic Risk class tactic Sprint tactic Plot Risk Items 17

18. 18© All rights reserved Requirements Map Requirements & Product Risks 1/3 Product Risks REQ 04 REQ 04 PR 02 PR 02 PR 05 PR 05 PR 01 PR 01 PR 04 PR 04 PR 03 PR 03 PR 08 PR 08 PR 07 PR 07 PR 06 PR 06 REQ 02 REQ 02 REQ 07 REQ 07 REQ 03 REQ 03 REQ 01 REQ 01 REQ 05 REQ 05 REQ 06 REQ 06 REQ 08 REQ 08 PR 09 PR 09 REQ 09 REQ 09 REQ 10 REQ 10 REQ 14 REQ 14 REQ 11 REQ 11 REQ 12 REQ 12 REQ 13 REQ 13 REQ 15 REQ 15

19. 19© All rights reserved REQ 14 REQ 14 REQ 11 REQ 11 REQ 12 REQ 12 REQ 13 REQ 13 REQ 15 REQ 15 Requirements Map Requirements & Product Risks 2/3 Product Risks REQ 04 REQ 04 PR 02 PR 02 PR 05 PR 05 PR 01 PR 01 PR 04 PR 04 PR 03 PR 03 PR 08 PR 08 PR 07 PR 07 PR 06 PR 06 REQ 02 REQ 02 REQ 07 REQ 07 REQ 03 REQ 03 REQ 01 REQ 01 REQ 05 REQ 05 REQ 06 REQ 06 REQ 08 REQ 08 PR 09 PR 09 REQ 09 REQ 09 REQ 10 REQ 10

20. © All rights reserved Map Requirements & Product Risks 3/3 20 Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 01●●●● REQ 02●●● REQ 03● REQ 04●● REQ 05●● REQ 06●● REQ 07●● REQ 08●● REQ 09●● REQ 10●●● REQ 11● REQ 12● REQ 13●●● REQ 14●●● REQ 15●●

21. © All rights reserved Gather Risk Items Gather & Order Requirements Gather Product Risks Plot Risk Items Map Requirements & Product Risks Calculate Risk Items Determine Impact & Likelihood Factors Score & Calculate Risk Items/Factors Strategy to tactic Risk class tactic Sprint tactic Calculate Risk Items 21

22. 22© All rights reserved Impact factors e.g. Frequency of Use Visibility Cost of failure … Determine Impact & Likelihood Factors Likelihood Factors e.g. Complexity Re-use Interfaces Dependencies Size … Interfaces None 1 - 2 > 2 Visibility Internal Partners Public

23. © All rights reserved Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 01●●●● REQ 02●●● REQ 03● REQ 04●● REQ 05●● REQ 06●● REQ 07●● REQ 08●● REQ 09●● REQ 10●●● REQ 11● REQ 12● REQ 13●●● REQ 14●●● REQ 15●● Score & Calculate Item/Factor 23 LIKELIHOOD PRODUCT RISK CLASS IMPACT Interfac es SizeRe-useTOTAL UsersVisibilit y Cost of Failure > 2> 5 SPNoneHigh PR 01 Medium Low< 5Internal< 1k None2 – 5 SPPartlyLow PR 02 Medium > 50Public< 1k 1 – 22 – 5 SPComplet ely Medium PR 03 Medium 5 - 50Internal< 1k 22> 5 SPPartlyHigh PR 04 High Medium> 50Partner s > 10k 22< 2 SPComplet ely Medium PR 05 Medium < 5Partner s < 1k 1 – 2< 2 SPComplet ely Low PR 06 Low < 5Internal1k – 10k 1 – 22 – 5 SPPartlyMedium PR 07 High 5 - 50Public> 10k None2 – 5 SPNoneLow PR 08 Low 5 – 50Internal< 1k None< 2 SPComplet ely Low PR 09 Low 5 - 50Internal< 1k PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 01???? REQ 02??? REQ 03? REQ 04?? REQ 05?? REQ 06?? REQ 07?? REQ 08?? REQ 09?? REQ 10??? REQ 11? REQ 12? REQ 13??? REQ 14??? REQ 15?? H M M H H H Product Risk Requirement PR X PR X PR X REQ XHMM HML MML M HMM M MH M MM H M M H M M H L M L MMM H L M LL LL M L

24. © All rights reserved Gather Risk Items Gather & Order Requirements Gather Product Risks Plot Risk Items Map Requirements & Product Risks Calculate Risk Items Determine Impact & Likelihood Factors Score & Calculate Risk Items/Factors Strategy to tactic Risk class tactic Sprint tactic Strategy to tactic 24

25. © All rights reserved HighMediumLow Check documentationInspectionWalkthroughPeer review UT test case reviewTest leadTester- ST test case reviewTest leadTester- Include test case in regression set Sprint + release Release- Automate test caseCodedUIRecord & Playback - Test execution on browsers Top 3Top 2Top 1 ………… Risk class tactic High = Effort * 3Medium = Effort * 2 Low = Effort * ½ 25

26. © All rights reserved Sprint Tactic – Scope Sprint 1 26 Product Risk Requirement PR 02 PR 04 PR 06 PR 08 PR 09 REQ 05 H M REQ 12 L REQ 13 M LL Rank # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

27. © All rights reserved Sprint Tactic – Scope Sprint 2 27 Product Risk Requirement PR 02 PR 03 PR 04 PR 09 REQ 03 M REQ 04 MM REQ 08 HL Rank # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

28. © All rights reserved28 Scrum bord To DoDoingDone REQ 3 REQ 4 REQ 8 SPRINT 2

29. © All rights reserved Insight into Product Risk & Requirement coverage 29 Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 05    REQ 12   REQ 13    REQ 03 M REQ 04 MM REQ 08 H L REQ 02 M MH REQ 06 M H REQ 07 M M REQ 09 M L REQ 01 M HMM REQ 10 MMM REQ 11 H REQ 14 LL REQ 15 M L  Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 05 REQ 12 REQ 13 REQ 03   REQ 04   REQ 08    REQ 02 M MH REQ 06 M H REQ 07 M M REQ 09 M L REQ 01 M HMM REQ 10 MMM REQ 11 H REQ 14 LL REQ 15 M L  Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 05 REQ 12 REQ 13 REQ 03 REQ 04 REQ 08 REQ 02    REQ 06    REQ 07    REQ 09    REQ 01 M HMM REQ 10 MMM REQ 11 H REQ 14 LL REQ 15 M L  Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 05 REQ 12 REQ 13 REQ 03 REQ 04 REQ 08 REQ 02 REQ 06 REQ 07 REQ 09 REQ 01    REQ 10   REQ 11   REQ 14 LL REQ 15 M L  Product Risk Requirement PR 01 PR 02 PR 03 PR 04 PR 05 PR 06 PR 07 PR 08 PR 09 REQ 05 REQ 12 REQ 13 REQ 03 REQ 04 REQ 08 REQ 02 REQ 06 REQ 07 REQ 09 REQ 01 REQ 10 REQ 11 REQ 14   REQ 15    

30. © All rights reserved Recap Agile RRBT Gather Risk Items Plot relationship between Risk Items Calculate and Classify Risk Items Differentiate the work Lessons Learned Analyses can only be done with the backlog in place. Plan to iterate and evolve your analyses in sync with the sprints. Don’t use the backlog as your only starting point for your analyses. The result of your analyses is essential in the planning meeting. Order your testing effort in the small. 30 B.W.F.P.M. BRONNEBERG TEST MANAGER / CONSULTANT REQUIREMENT & QUALITY MANAGEMENT BRAM.BRONNEBERG@CGI.COM WWW.BRAMBRONNEBERG.NL @BRAMBRONNEBERG +31 6 55 11 35 10 B.W.F.P.M. BRONNEBERG TEST MANAGER / CONSULTANT REQUIREMENT & QUALITY MANAGEMENT BRAM.BRONNEBERG@CGI.COM WWW.BRAMBRONNEBERG.NL @BRAMBRONNEBERG +31 6 55 11 35 10