1. Thomas L. Gilchrist tomg@tomgtomg.com Testing Basics Set 4: Strategies & Metrics By Thomas L. Gilchrist, 2009

2. Thomas L. Gilchrist tomg@tomgtomg.com What are the Tradeoffs …changes in one require compensation in others...

3. Thomas L. Gilchrist tomg@tomgtomg.com Tradeoffs Schedule Cost/Resources Scope Technology Quality / Critical Success Factors

4. Thomas L. Gilchrist tomg@tomgtomg.com How much testing? …risk!...

5. Thomas L. Gilchrist tomg@tomgtomg.com Testing and Risk The degree of risk increases The degree of testing

6. Thomas L. Gilchrist tomg@tomgtomg.com High Low High Low Where are you now? Level of testing Confidence in process TREND Testing and Risk

7. Thomas L. Gilchrist tomg@tomgtomg.com Testing Goals Testing is about finding errors. If you don’t find any errors, testing is a waste of time.

8. Thomas L. Gilchrist tomg@tomgtomg.com Testing and Risk Maturity of the development process The cost to find an error/bug

9. Thomas L. Gilchrist tomg@tomgtomg.com Definitions Requirement –A condition or capability needed by a user to solve a problem or achieve an objective –A condition or capability that must be met or possessed by a system …to satisfy a contract, standard, specification or other formally imposed document.

10. Thomas L. Gilchrist tomg@tomgtomg.com Test Terms Test Categories –Unit Testing –Integration Testing –System Testing –Acceptance Testing –Regression Testing Static Testing Dynamic Testing

11. Thomas L. Gilchrist tomg@tomgtomg.com The Test Process

12. Thomas L. Gilchrist tomg@tomgtomg.com The Testing Process Step 1 - Set Test Objectives Step 2 - Develop Test Plan Step 3 - Execute Tests Step 4 - Summarize and Report Results

13. Thomas L. Gilchrist tomg@tomgtomg.com General Workbench Input Source Materials Process Product ENTRYENTRY EXITEXIT Standards, templates, rules and checklists Do Work Rework Testing

14. Thomas L. Gilchrist tomg@tomgtomg.com Testing Workbench Test Process Products for Test Problems Tested Products ENTRYENTRY EXITEXIT Test Toolbox Test QC Test Measures

15. Thomas L. Gilchrist tomg@tomgtomg.com The Software Test Process Step 1 - Develop Test Objectives Task 1 - Organize Test Team Task 2 - Perform Risk Assessment Task 3 - Set Test Objectives Task 4 - Quality Control Step 2 - Develop Test Plan Task 1 - Define Business Functions Task 2 - Define Structural Functions Task 3 - Define Tests Task 4 - Create Function/Test Matrix Task 5 - Admin Test Requirements Task 6 - Formalize System Test Plan Task 7 - System Test Plan QC Task 8 - Develop Unit Test Plan Task 9 - Unit Test Plan QC Step 3 - Execute Tests Task 1 - Select Test Tools Task 2 - Develop Test Cases Task 3 - Execute Tests Task 4 - Quality Control Step 4 - Summarize and Report Results Task 1 - Record Defects Task 2 - Perform Data Reduction Task 3 - Develop Findings Task 4 - Formalize Report Task 5 - Quality Control

16. Thomas L. Gilchrist tomg@tomgtomg.com Test Process “V” Diagram Business Need Define Requirements Design System Build System Verify Acceptance Test System Test Integration Test Unit Test Verify Validate Validates Test Planning Test Objectives Test Execution (Static) Test Execution (Dynamic) & Test Reporting

17. Thomas L. Gilchrist tomg@tomgtomg.com The SEI CMM

18. Thomas L. Gilchrist tomg@tomgtomg.com Immature Vs Mature Organization ImmatureMature Heroic Efforts of Dedicated Team Repeating Proven Methods of an Organization Not ManagedManaged People Management Technology Process Not Fully Utilized Organization al Focus No "Memory", Improvised Repeatable, Followed, Predictable From: SEI CMM/Bill Curtis, et al.

19. Thomas L. Gilchrist tomg@tomgtomg.com CMM Architecture LevelKey Process Areas 1 Initial Heroes* 5 Optimizing System CQI Defect Prevention Technology Innovation Process Change Management 4 Managed System Measurement Process Measurement and Analysis Quality Management 3 Defined Engineering Technical Practices Organizational Process Focus Organizational Process Definition Peer Reviews Training Program Intergroup Coordination Software Product Engineering Integrated Software Management 2 Repeatable Project Management Software Project Planning Software Project Tracking Software Subcontract Management Software Quality Assurance Software Configuration Management Requirements Management Process Focus From: SEI CMM/Bill Curtis, et al.

20. Thomas L. Gilchrist tomg@tomgtomg.com Taking Action Based On Data

21. Thomas L. Gilchrist tomg@tomgtomg.com Measurement What’s you least favorite measurement? What’s your favorite measurement?

22. Thomas L. Gilchrist tomg@tomgtomg.com Designing a Metric Program GQMGQM * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

23. Thomas L. Gilchrist tomg@tomgtomg.com Designing a Metric Program Goal Q M * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

24. Thomas L. Gilchrist tomg@tomgtomg.com Designing a Metric Program Goal Question M * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

25. Thomas L. Gilchrist tomg@tomgtomg.com Designing a Metric Program Goal Question Metric * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

26. Thomas L. Gilchrist tomg@tomgtomg.com Designing a Metric Program Goal Question Metric * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

27. Thomas L. Gilchrist tomg@tomgtomg.com Understanding Metric Data Data is not information! DataInformation From: “Understanding Variation”, D.J. Wheeler

28. Thomas L. Gilchrist tomg@tomgtomg.com Understanding Metric Data Data is not information! DataInformationAnalysis From: “Understanding Variation”, D.J. Wheeler

29. Thomas L. Gilchrist tomg@tomgtomg.com Key Variation Concepts Variation exists everywhere Two types –Common causes (noise) –Special case (signal) Variation, waste, & predictability. 94% of all problems in an organization are due to common causes and only management can fix them. (Deming, “Out of Crisis”)

30. Thomas L. Gilchrist tomg@tomgtomg.com Run Charts Metric Time

31. Thomas L. Gilchrist tomg@tomgtomg.com Specification Targets Upper Specification Limit Lower Specification Limit Metric Time Voice of the customer

32. Thomas L. Gilchrist tomg@tomgtomg.com Shewhart’s Control Charts Every data set has noise. “Noise” confuses and clouds single value comparisons. Some data sets contain signal (special causes). Control charts separate potential signals from noise.

33. Thomas L. Gilchrist tomg@tomgtomg.com Control Chart Metric Time Voice of the process

34. Thomas L. Gilchrist tomg@tomgtomg.com Control Chart Metric Time Voice of the process Upper Control Limit Lower Control Limit

35. Thomas L. Gilchrist tomg@tomgtomg.com Signal and Noise Upper Control Limit Lower Control Limit Metric Time Voice of the process "Special Cause" "Common Cause" Region Data excursions

36. Thomas L. Gilchrist tomg@tomgtomg.com Serial Correlation Metric Time Voice of the process Upper Control Limit Lower Control Limit

37. Thomas L. Gilchrist tomg@tomgtomg.com Recommendations Improve measurement systems –Commit to measurement, not guessing –Devise and use common measurement tools and definitions –Train data collectors on metric definitions Reduce process variation Improve process results

38. Thomas L. Gilchrist tomg@tomgtomg.com Variation In Software In software, when one process has wide variation, more than likely, they all have have wide variation. High precision in software measurement is unnecessary at this time.

39. Thomas L. Gilchrist tomg@tomgtomg.com Suboptimization Suboptimization occurs when one doesn’t think of the total system...it is about dependencies.

40. Thomas L. Gilchrist tomg@tomgtomg.com X+Y=1 Suboptimization X+Y=1 Suboptimization occurs when one doesn’t think of the total system...it is about dependencies.

41. Thomas L. Gilchrist tomg@tomgtomg.com Suboptimization X is big and Y is big = goodness How do we allocate resources between x and Y? What are some examples? Suboptimization occurs when one doesn’t think of the total system...it is about dependencies. X+Y=1