1. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.1 Selecting an Appropriate Software Testing Method Ibrahim K. El-Far Florida Institute of Technology

2. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.2 Overview Needed research areas Test method evaluation issues What to evaluate Hidden misconceptions in comparisons Factors influencing evaluation

3. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.3 Categories of Quality Activities Going into the product Good process Documentation Formal analysis Design reviews Code Inspections After the fact Program proof Fault injection Mutation analysis Testing

4. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.4 Variety of Testing Styles Scenario Exploratory model-based Stochastic Control flow Functional Data flow Clear box Partition Black box Random Boundary Value Risk-based Requirements-based

5. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.5 Needed Research Areas How to evaluate new test method Assess worth, goals, activities Estimate how well it integrates into existing practices and tools How to evaluate existing methods Study thoroughness and cost Determine where improvements can be made How to put methods to work in combination

6. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.6 Test Method Evaluation Issues What is it about the testing method that we want to evaluate? Does the evaluation involve any entity other than the method itself? Is there a precise set of measures that describe what we want to evaluate? Are there uncontrollable factors affecting the evaluation? What is the scope of the results of the evaluation?

7. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.7 What to Evaluate Effectiveness – Is the method producing the right results? Efficiency – Is the method wasting as little resources as possible? Human elements – Is the method dependent on skills, expertise, knowledge, and personal communication?

8. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.8 Comparing Effectiveness Number of defects found Delivered reliability Detecting defects of a specific class Satisfying the adequacy criteria Power of finding faults

9. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.9 Comparing Cost Size of test suites Speed of producing test suites Return of investments Bugs Reliability Information

10. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.10 Comparing Human Element Degree of human action involvement Dependency on human skills Dependency on domain expertise Amount of needed on preparation

11. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.11 Factors Influencing Evaluation The Nature of Programs The Nature of The Method Circumstances Surrounding Development and Test

12. (c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe 2002.12 Contact Information Ibrahim K. El-Far ielfar@acm.org http://www.testingresearch.com/ Department of Computer Sciences, Florida Institute of Technology, 150 W. University Blvd., Melbourne, Florida 32901, USA.