(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Selecting an Appropriate Software Testing Method Ibrahim K. El-Far Florida Institute of Technology

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Overview Needed research areas Test method evaluation issues What to evaluate Hidden misconceptions in comparisons Factors influencing evaluation

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Categories of Quality Activities Going into the product Good process Documentation Formal analysis Design reviews Code Inspections After the fact Program proof Fault injection Mutation analysis Testing

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Variety of Testing Styles Scenario Exploratory model-based Stochastic Control flow Functional Data flow Clear box Partition Black box Random Boundary Value Risk-based Requirements-based

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Needed Research Areas How to evaluate new test method Assess worth, goals, activities Estimate how well it integrates into existing practices and tools How to evaluate existing methods Study thoroughness and cost Determine where improvements can be made How to put methods to work in combination

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Test Method Evaluation Issues What is it about the testing method that we want to evaluate? Does the evaluation involve any entity other than the method itself? Is there a precise set of measures that describe what we want to evaluate? Are there uncontrollable factors affecting the evaluation? What is the scope of the results of the evaluation?

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe What to Evaluate Effectiveness – Is the method producing the right results? Efficiency – Is the method wasting as little resources as possible? Human elements – Is the method dependent on skills, expertise, knowledge, and personal communication?

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Comparing Effectiveness Number of defects found Delivered reliability Detecting defects of a specific class Satisfying the adequacy criteria Power of finding faults

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Comparing Cost Size of test suites Speed of producing test suites Return of investments Bugs Reliability Information

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Comparing Human Element Degree of human action involvement Dependency on human skills Dependency on domain expertise Amount of needed on preparation

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Factors Influencing Evaluation The Nature of Programs The Nature of The Method Circumstances Surrounding Development and Test

(c) Copyright 2002 Ibrahim K. El-Far. All rights reserved. Presented at Quality Week Europe Contact Information Ibrahim K. El-Far Department of Computer Sciences, Florida Institute of Technology, 150 W. University Blvd., Melbourne, Florida 32901, USA.