CEN 5070 – Software V&V What is Software Testing © , Dr. E.L. Jones

9/2003Testing Concepts2 Purpose This module presents the basic concepts of software testing. We present several aspects of software testing, and a give a generic lifecycle. "… arbitrarily selected test set... results in inefficient testing, leaving some functions untested while performing redundant testing of others." Darlene Mackay, Quality Consultants Unlimited

9/2003Testing Concepts3 Agenda What is Testing Testing Lifecycle A Framework for Testing A Complete Example

9/2003Testing Concepts4 WHAT IS TESTING? Executing software to determine whether it meets its specification Executing a software element to discover defects or anomalies Executing software to evaluate its behavior in different environments Inspecting software element code to discover defects or anomalies.

9/2003Testing Concepts5 TESTING IS HARD Exhaustive testing is impossible Software is complex Specifying requirements is hard Testing is a fallible, human process Consequences of failure not always understood in advance

9/2003Testing Concepts6 WORKING SMART Test small pieces Use proven techniques Use tools to advantage Start early Cost of errors grows exponentially with time they go undetected

9/2003Testing Concepts7 Agenda What is Testing Testing Lifecycle A Framework for Testing A Complete Example

9/2003Testing Concepts8 A TESTING LIFECYCLE Analysis Design Implementation Execution Evaluation Specification Test Strategy/Plan Test Script, Data, Driver Defect Data Problem Reports Test Results Test Cases

9/2003Testing Concepts9 TESTING vs DEVELOPMENT Work broken into stages Require similar skills Require management oversight Produce artifacts used to assess quality of the work Test artifacts are overhead, not part of delivered system.

9/2003Testing Concepts10 TEST ARTIFACTS Strategy/Plan (must know techniques) Test cases – central artifacts Test machinery to apply test cases Data showing results Reports that interpret test results List of problems/defects to be fixed.

9/2003Testing Concepts11 TESTING IS HIERARCHICAL Unit testing – smallest components Aggregate testing – assemblies of units Integration testing – combinations of assemblies System testing – system built from assemblies

9/2003Testing Concepts12 TESTING OBJECTIVES To demonstrate conformance To find defects To measure behavior in specific environments Load / stress testing Performance testing To measure trustworthiness/reliability

9/2003Testing Concepts13 Agenda What is Testing Testing Lifecycle A Framework for Testing A Complete Example

9/2003Testing Concepts14 REQUIRED ELEMENTS OF A TESTING METHODOLOGY? Specification for the software Process for designing test cases Repeatable process for designing, running and evaluating tests Accountable artifacts of testing activities Economical use of human, time and computing resources

9/2003Testing Concepts15 Agenda What is Testing Testing Lifecycle A Framework for Testing A Complete Example

9/2003Testing Concepts16 BLACK-BOX TESTING Testing based on the specification rather than the implementation. Specification defines the expected response(s) to stimuli Software under Test StimuliResponse(s)

9/2003Testing Concepts17 BLACK-BOX TECHNIQUES Functional testing -- tests the behavior of the software. Boundary testing -- tests behavior at the lower/upper bounds of input values Random testing -- tests using randomly generated stimuli Intuitive (ad hoc) testing -- error guessing

9/2003Testing Concepts18 FUNCTIONAL TEST DESIGN METHODOLOGY Specification Identify behaviors Develop test cases Write test script

9/2003Testing Concepts19 EXAMPLE A (1) Specification Compute pay for an hourly employee, given the number of hours worked and the hourly pay rate. Compute overtime at 1.5 times hourly rate for hours in excess of 40. Software under Test HoursPay Rate

9/2003Testing Concepts20 EXAMPLE A (2) Identify Behaviors Case 1: No overtime (Hours <= 40) Expect Pay = Hours * Rate Case 2: Overtime (Hours > 40) Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)

9/2003Testing Concepts21 EXAMPLE A (3) Create Test Cases Case 1: No overtime (Hours <= 40) Use Rate = 10, Hours = 30 Expect Pay = Hours * Rate = 300 Case 2: Overtime (Hours > 40) Use Rate = 10, Hours = 50 Expect Pay = 40*Rate+1.5*Rate*(Hours - 40) = 550

9/2003Testing Concepts22 EXAMPLE A (4) Write Test Script Step StimuliExpected Response HoursRatePay =

9/2003Testing Concepts23 A MORE COMPLEX EXAMPLE (B) Increased number of behaviors Use of decision table to document behaviors Test case generation from decision table

9/2003Testing Concepts24 EXAMPLE B (1) Specification Compute pay for employee, given the number of hours worked and the hourly pay rate. For hourly employees (rate = 30) are paid for exactly 40 hours.

9/2003Testing Concepts25 EXAMPLE B (2) Identify Behaviors Case 1: Hourly AND No overtime (Rate < 30) & (Hours <= 40) Expect Pay = Hours * Rate Case 2: Hourly AND Overtime (Rate 40) Expect Pay = 40*Rate+1.5*Rate*(Hours - 40) Case 3: Salaried (Rate >= 30) Expect Pay = 40 * Rate

9/2003Testing Concepts26 DECISION TABLE Condition c1: Rate < 30 | Y Y N N c2: Hours <= 40 | Y N Y N Action a1: Pay = Straight time | X a2: Pay = Overtime | X a3: Pay = Professional | X X Columns define Behaviors

9/2003Testing Concepts27 EXAMPLE B (3) Create Test Cases One test case per column of decision table Case 1: Hourly, No Overtime Case 2: Hourly, Overtime Case 3: Salaried, No Extra Hours Case 4: Salaried, Extra Hours Order the test cases by column

9/2003Testing Concepts28 EXAMPLE B (4) Write Test Script Step StimuliExpected Response HoursRatePay =

9/2003Testing Concepts29 RULES -- DECISION TABLES Condition c1: Rate < 30 | Y Y N N c2: Hours <= 40 | Y N Y N Action a1: Pay = Straight time | X a2: Pay = Overtime | X a3: Pay = Professional | X X Use X to select action(s) Elementary conditions Use 'Y', 'N', '-' or space

9/2003Testing Concepts30 NEXT TIME Test Case Generation Strategies Documenting the Test Specification Techniques