1. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 37 Testing and Metrics

2. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 2 Outline Testing Testing Metrics

3. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 3 What is Testing? I.e., executing the software under conditions similar to actual operation Testing is Finding Defects by Exercising the Software

4. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 4 Testing for Defect Detection Testing is one of the principal methods of defect detection Testing is actually performed at every stage of software development if modern software engineering methods are used Each stage also involves planning and preparation for tests to be performed later

5. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 5 Why Do Testing? To demonstrate product features? – This is not the purpose of testing. This is for demos To show that there are no errors? – This is impossible To show that errors are unlikely? – Reasonable, if testing is done properly … continued

6. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 6 Why Do Testing? (continued) To show that the software works correctly? – OK, under test conditions only, if testing is done properly To find errors? – Should be the main objective

7. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 7 Standard Terms for Software Decomposition System - a functioning device of which software is a part. – Examples: an airplane; a personal computer; a payroll system; a financial system Product - an individual software element that is developed, tested, and sold separately – Examples: a radar controller; a spreadsheet program; perhaps a payroll system or part of one

8. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 8 Standard Terms for Software Decomposition (continued) Component - a part of a product that can be specified and tested independently – Example: an input/output package; a data analysis program; an object or class Module or Unit - the smallest element that can be tested independently – Example: a subroutine; a lowest level object

9. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 9 Standard Terms 1 for Software Decomposition -- Pictorial Personal Computer System UnitPeripheralsMonitorSoftware Data Base Word Processor Operating System I/O Pkg Spell Check Subroutine System: Sub- Systems: Products : Components : Modules: 1 Note that these are arbitrary terms

10. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 10 Tying the Process Stages to the Product Terminology - Page 1 System Architecting and System Design produce the following: – Specification of system requirements – Identification of subsystems and products – Allocation of specifications (requirements) to different subsystems and products Software Requirements Analysis produces: – Specification of the software requirements for each product

11. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 11 Tying the Process Stages to the Product Terminology - Page 2 Software Design produces: – Identification of the software components and interfaces – Allocation of software specifications/requirements to components – Identification of the sub-components and modules – Allocation of specifications/requirements to modules

12. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 12 Tying the Process Stages to the Product Terminology - Page 3 Code and Unit Test produces: – Code for modules – Test code for modules – (Unit) tests performed on modules Software Product Integration and Test produces: – Test code for components and product – (Integration) tests performed on modules and product

13. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 13 Tying the Process Stages to the Product Terminology - Page 4 System Integration and Test produces: – Integration of software and hardware products – Test sets and/or code for the system or subsystems – (System integration) tests performed on subsystems and on the overall system

14. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 14 “Testing” at Requirements Stage of Software Development Requirements Simulations and Analyses Analyze Requirements Construct Requirements Model Specify Requirements Simulate Requirements Model inconsistencies, errors, etc. inconsistencies incompleteness etc. To Design Stage Requirements Specification Test Requirements

15. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 15 “Testing” at Design Stage of Software Development Design Simulations and Analyses Synthesize Design Construct Design Model Specify Design Simulate and/or Compile the Design Model inconsistencies, errors, etc. inconsistencies incompleteness performance problems To Coding Stage Design Specification Test Cases

16. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 16 Implementation (Code and Unit Test) Develop Code To Integration Stage Develop Test Code & Procedures Test Cases (Sample Inputs & Expected Outputs) code test code & procedures Peer Reviews (walkthroughs or inspections) Perform Unit Tests errors inconsistencies incompleteness performance problems

17. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 17 Integration & System Test Module 1 Module 2 Module 3 Module 4 Module 5 Integrate and Test (*) (*) Test is similar to unit test... To further integration and test steps integrated component integrated component Test Cases Test Code

18. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 18 Different Testing at Each Stage We have seen that the nature of testing differs at each stage But there is another dimension that relates to the specific focus of testing at each stage At each stage we test different things This can be seen from the V diagram (next slide)

19. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 19 The V Diagram View of Testing System Analysis SW Rqmts Analysis Hi Level SW Design Detailed SW Design Rqmts Allocation Coding Unit Test Component Integ. &Test Product Integ. &Test Sub-Syst. Integ. &Test System Int. &Test Unit Test Cases (White Box) Component Level Test Cases (White Box) Product Level Test Cases (Black Box) System Level Test Cases Sub-System Level Test Cases

20. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 20 Kinds of Tests White Box Tests (Glass Box Tests) – These focus on testing correct structure and design of the software – Weak on assuring that requirements are met Black Box Tests – Focus on testing the functionality of the software relative to its requirements – Weak on assuring that structure is correct Undefined variables Initialization errors alternative ways to implement the design

21. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 21 Ticking Box Tests These kinds of tests result from such things as: – Poor test planning – Inadequate time or resources for testing – Testing to prove it works rather than to find defects Tests that explode when you least want them to

22. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 22 Testing Metrics A number of things can be measured about tests These are mostly process metrics

23. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 23 Testing Metrics - Conformance You can measure the conformance of all test plans, test cases, etc. to standards or conventions This helps you determine whether they have a high probability of being effective, complete, etc. This is similar to the way you might measure code complexity or other attributes

24. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 24 Testing Metrics - Effectiveness You can measure test results (what percentage of tests find errors; what percentage fail to find errors that later show up) This can help you improve the effectiveness of your testing methods and techniques

25. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 25 Testing Metrics - Execution You can measure whether tests are actually run / run properly These enable you to manage and control the process and the quality of the product

26. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 26 In Summary Effective testing is a comprehensive activity that involves all phases of the process Many things can be measured to help improve the effectiveness of a test approach

27. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M37 8/20/2001Slide 27 END OF MODULE 37