1. Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu

2. Copyright (c) 2003 Howard E. Dow2 Agenda Audience - Project - Terminology What the team did Results and data Benefits, pitfalls and how to Conclusions

3. Copyright (c) 2003 Howard E. Dow3 Car Cah Parked Pahked Rs ???? Idea Idear

4. Copyright (c) 2003 Howard E. Dow4 Target Audience Engineers Project leaders Ideally Trained on inspections Some inspection experience

5. Copyright (c) 2003 Howard E. Dow5 Project Call center system Functional tests Goals Defect free test scripts On time Increase skill level

6. Copyright (c) 2003 Howard E. Dow6 Skill levels

7. Copyright (c) 2003 Howard E. Dow7 Terminology Inspection = Software Inspection = Formal Inspection = “Fagan Style” Inspection Reviews Walk-Throughs Any technique

8. Copyright (c) 2003 Howard E. Dow8 Terminology Inspection characteristics Goals Rules, Roles, Responsibilities Process

9. Copyright (c) 2003 Howard E. Dow9 Goals of Inspections Find defects Collect data Archive data Use data Learn

10. Copyright (c) 2003 Howard E. Dow10 Etiquette, Customs and Rules Focus on the product Be prepared Open mind Language

11. Copyright (c) 2003 Howard E. Dow11 Roles and Responsibilities ModeratorRecorder Reader Author Management Inspector

12. Copyright (c) 2003 Howard E. Dow12 Planning Product Overview Preparation Inspection Rework Data analysis and storage Re-inspect (optional) Inspection Process

13. Copyright (c) 2003 Howard E. Dow13  Audience - Project - Terminology  What the team did  Results and data  Benefits, pitfalls and how to  Conclusions Agenda

14. Copyright (c) 2003 Howard E. Dow14 ModeratorRecorder ReaderAuthor Inspector Manager Roles and Responsibilities

15. Copyright (c) 2003 Howard E. Dow15 Product Overview Planning Preparation Inspection Rework Data analysis and storage Re-inspect (optional) Allowed suggestions Time not collected Inspection Process Used

16. Copyright (c) 2003 Howard E. Dow16 Summary of key differences Omit product overview Involved management Suggesting fixes allowed Strong bias to re-examine after rework by another team member Omission - collecting preparation time

17. Copyright (c) 2003 Howard E. Dow17 Ready to go 17 Scripts 3-6 inspectors

18. Copyright (c) 2003 Howard E. Dow18 Data Collected Script identifier Author Date completed Date inspected Size Defect: number, location, description Inspection time Inspectors

19. Copyright (c) 2003 Howard E. Dow19  Audience - Project - Terminology  What the team did  Results and data  Benefits, pitfalls and how to  Conclusions Agenda

20. Copyright (c) 2003 Howard E. Dow20 Results No operational defects found in inspected test scripts On time delivery Increased skill level Defects

21. Copyright (c) 2003 Howard E. Dow21 Summary Data - 1 Totals Scripts inspected: 17 LOC inspected: 12,949 Defects found: 544 Time spent: 24.2 hours 75 hours

22. Copyright (c) 2003 Howard E. Dow22 Summary Data - 2

23. Copyright (c) 2003 Howard E. Dow23 Summary Data - 3 Calculated Defect density: 42 Defects/KLOC Inspection rate: 535 LOC/hour 173 LOC/hour Defect find rate: 22.5 Defects/hour 7 Defects/hr

24. Copyright (c) 2003 Howard E. Dow24 Summary Data - 4 Severity typeQuantityPercent of total High152.8% Medium20938.4% Low32058.8% Total544100%

25. Copyright (c) 2003 Howard E. Dow25

26. Copyright (c) 2003 Howard E. Dow26

27. Copyright (c) 2003 Howard E. Dow27

28. Copyright (c) 2003 Howard E. Dow28

29. Copyright (c) 2003 Howard E. Dow29

30. Copyright (c) 2003 Howard E. Dow30

31. Copyright (c) 2003 Howard E. Dow31 Observation

32. Copyright (c) 2003 Howard E. Dow32 Observation

33. Copyright (c) 2003 Howard E. Dow33 Observation ???

34. Copyright (c) 2003 Howard E. Dow34 Observation

35. Copyright (c) 2003 Howard E. Dow35 Excellent! Observation

36. Copyright (c) 2003 Howard E. Dow36  Audience - Project - Terminology  What the team did  Results and data  Benefits, pitfalls and how to  Conclusions Agenda

37. Copyright (c) 2003 Howard E. Dow37 Benefits No operational defects reported On time delivery Value of process shown Some increase in expertise

38. Copyright (c) 2003 Howard E. Dow38 Pitfalls Must be “Sloppy” data collection Need for more training

39. Copyright (c) 2003 Howard E. Dow39 How to Find a “critical” project Suggest doing something different Keep it simple - be flexible Propose “Let’s put more eyes on the code.” “Let’s inspect the code.”

40. Copyright (c) 2003 Howard E. Dow40  Audience - Project - Terminology  What the team did  Results and data  Benefits, pitfalls and how to  Conclusions Agenda

41. Copyright (c) 2003 Howard E. Dow41 Conclusions Test scripts can be inspected Share your results Other testers Developers Useful as a learning tool Avoid unrealistic expectations

42. Copyright (c) 2003 Howard E. Dow42 References The original article on software inspections. “Design and Code Inspections to Reduce Errors in Program Development”, Michael Fagan, IBM Systems Journal, Volume 15, No. 3, 1976. This may be available as reprint at: http://www.research.ibm.com/journal/si/382/fagan.pdf Handbook of Walkthroughs, Inspections, and Technical Reviews, Daniel P. Freedman and Gerald M. Weinberg, 3 rd Edition, Dorset House, 1990 Software Inspection, Tom Gilb and Dorothy Graham, Addison-Wesley, 1990

43. Copyright (c) 2003 Howard E. Dow43 Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu Thank you for your time. Questions please