1. ECE 355: Software Engineering
CHAPTER 11
Part II

2. Course outline Unit 1: Software Engineering Basics
Unit 2: Process Models and Software Life Cycles
Unit 3: Software Requirements
Unit 4: Unified Modeling Language (UML)
Unit 5: Design Basics and Software Architecture
Unit 6: OO Analysis and Design
Unit 7: Design Patterns
 Unit 8: Testing and Reliability
Unit 9: Software Engineering Management and Economics

3. Course Outline Introduction to software engineering
Requirements Engineering
Design Basics
Traditional Design
OO Design
Design Patterns
Software Architecture
Design Documentation
Verification & Validation
Software Process Management and Economics

4. These slides are based on:
Lecture slides by Ian Summerville, see
ECE355 Lecture slides by Sagar Naik
Lecture Notes from Bernd Bruegge, Allen H. Dutoit “Object-Oriented Software Engineering – Using UML, Patterns and Java”

5. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

6. Static and dynamic V&V Static verification Requirements specification
Architecture
Detailed
design
Implementation V1
Implementation V1
Prototype
Dynamic
V &V
Special cases
Executable specifications
Animation of formal specs

7. Program testing Can reveal the presence of errors NOT their absence
Only exhaustive testing can show a program is free from defects. However, exhaustive testing for any but trivial programs is impossible
A successful test is a test which discovers one or more errors
Should be used in conjunction with static verification
Run all tests after modifying a system
©Ian Sommerville 1995

8. Testing in the V-Model Acceptance Requirements test Customer Developer
Architectural
Design
System test
Run tests
Integration test
Write tests
Detailed
Design
Functional
(BB)
Structural
(WB)
Module
implementation
Unit test

9. Testing stages Unit testing Integration testing System testing
Testing of individual components
Integration testing
Testing to expose problems arising from the combination of components
System testing
Testing the complete system prior to delivery
Acceptance testing
Testing by users to check that the system satisfies requirements. Sometimes called alpha testing

10. Types of testing Statistical testing Defect testing
Tests designed to reflect the frequency of user inputs. Used for reliability estimation.
Covered later in section on Software reliability.
Defect testing
Tests designed to discover system defects.
A successful defect test is one which reveals the presence of defects in a system.
©Ian Sommerville 1995

11. Some Terminology Failure Error Fault
A failure is said to occur whenever the external behavior does not conform to system spec.
Error
An error is a state of the system which, in the absence of any corrective action, could lead to a failure.
Fault
An adjudged cause of an error.

12. Some Terminology fault, bug, error, defect It is there in the program…
Program state
Error
Observed
Failure

13. Testing Activities Unit T est Unit T est Integration Functional Test
Requirements
Analysis
Document
Subsystem
Code
Unit
Requirements
Analysis
Document
System
Design
Document T
est
Tested
Subsystem
User
Manual
Subsystem
Code
Unit T
est
Tested
Subsystem
Integration
Functional
Test
Test
Functioning
System
Integrated
Subsystems
Tested Subsystem
Subsystem
Code
Unit T
est
All tests by developer

14. Testing Activities continued
Client’s
Understanding
of Requirements
Global
Requirements
User
Environment
Validated
System
Accepted
System
Functioning
System
Performance
Acceptance
Installation
Test
Test
Test
Usable
System
Tests by client
Tests by developer
User’s understanding
System in
Use
Tests (?) by user

15. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

16. Testing and debugging
Defect testing and debugging are distinct processes
Defect testing is concerned with confirming the presence of errors
Debugging is concerned with locating and repairing these errors
Debugging involves formulating a hypothesis about program behaviour then testing these hypotheses to find the system error
©Ian Sommerville 1995

17. Debugging Activities Locate error & fault Design fault repair
Repair fault
Re-test
program
©Ian Sommerville 1995

18. Testing Activities Test result Identify
Test conditions (“What”): an item or event to be verified.
How the “what” can be tested: realization
Design
Build test cases (imp. scripts, data)
Build
Run the system
Execute
Test case outcome with
expected outcome
Compare
Test result

19. Testing Activities Test condition
What: Descriptions of circumstances that could be examined (event or item).
Categories: functionality, performance, stress, robustness…
Derive
Using testing techniques (to be discussed)
(Refer to the V-Model)

20. Testing Activities Design test cases: the details Input values
Expected outcomes
Things created (output)
Things changed/updated  database?
Things deleted
Timing …
Known
Unknown (examine the first actual outcome)
Environment prerequisites: file, net connection …

21. Testing Activities Build test cases (implement)
Implement the preconditions (set up the environment)
Prepare test scripts (may use test automation tools)
Structure of a test case
Simple linear Tree
(I, EO) {(I1, EO1), (I2, EO2), …} I
EO1
EO2

22. Testing Activities Scripts contain data and instructions for testing
Comparison information
What screen data to capture
When/where to read input
Control information
Repeat a set of inputs
Make a decision based on output
Testing concurrent activities
Sync.: when to enter input

23. Testing Activities Compare (test outcomes, expected outcomes)
Simple/complex (known differences)
Different types of outcomes
Variable values (in memory)
Disk-based (textual, non-textual, database, binary)
Screen-based (char., GUI, images)
Others (multimedia, communicating apps.)

24. Testing Activities Compare: actual output == expected output?? Yes No
Pass (Assumption: Test case was “instrumented.”) No
Fail (Assumption: No error in test case, preconditions)

25. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

26. Goodness of test cases Exec. of a test case against a program P
Covers certain requirements of P;
Covers certain parts of P’s functionality;
Covers certain parts of P’s internal logic.
 Idea of coverage guides test case selection.

27. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

28. Black-box Testing
Focus: I/O behavior. If for any given input, we can predict the output, then the module passes the test.
Almost always impossible to generate all possible inputs ("test cases")
Goal: Reduce number of test cases by equivalence partitioning:
Divide input conditions into equivalence classes
Choose test cases for each equivalence class. (Example: If an object is supposed to accept a negative number, testing one negative number is enough)

29. Black-box Testing (Continued)
Selection of equivalence classes (No rules, only guidelines):
Input is valid across range of values. Select test cases from 3 equivalence classes:
Below the range
Within the range
Above the range
Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes:
Valid discrete value
Invalid discrete value
Another solution to select only a limited amount of test cases:
Get knowledge about the inner workings of the unit being tested => white-box testing

30. Equivalence partitioning
©Ian Sommerville 1995

31. Equivalence partitioning
Partition system inputs and outputs into ‘equivalence sets’
If input is a 5-digit integer between and 99999, equivalence partitions are <10000, and >99999
If you can predict that certain set of inputs will be treated differently in processing from another one, put them into separate partitions, e.g., Canadian and US addresses (will validate state/province), addresses in other countries (no state validation), etc.
Two test case selection strategies
Choose one random test case from each partition
Choose test cases at the boundary of the partitions
09999, 10000, 99999,
©Ian Sommerville 1995

32. Equivalence partitions
09999
10000
50000
99999
100000
Less than 10000
Between and 99999
More than 99999
©Ian Sommerville 1995

33. Search routine specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
©Ian Sommerville 1995

34. Strategy for input partitions (I) (General)
Inputs which conform to the pre-conditions
Inputs where a pre-condition does not hold
Inputs derived from the post-condition
Inputs where the key element is a member of the array
Inputs where the key element is not a member of the array
©Ian Sommerville 1995 [modified]

35. Strategy for input partitions (II) (Arrays)
Test software with arrays which have only a single value
Test with arrays of zero length (if allowed by programming language)
Use arrays of different sizes in different tests
Derive tests so that the first, middle and last elements of the array are accessed
©Ian Sommerville 1995

36. Search routine - input partitions
No value
Not in array
©Ian Sommerville 1995 [modified]

37. Search routine - test cases() 1
??,??
©Ian Sommerville 1995 [modified]

38. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

39. White-box Testing
Statement Testing (Algebraic Testing): Test single statements (Choice of operators in polynomials, etc)
Loop Testing:
Cause execution of the loop to be skipped completely. (Exception: Repeat loops)
Loop to be executed exactly once
Loop to be executed more than once
Path testing:
Make sure all paths in the program are executed
Branch Testing (Conditional Testing): Make sure that each possible outcome from a condition is tested at least once
if ( i =
TRUE) printf("YES\n");
else
printf("NO\n");
Test cases: 1) i = TRUE; 2) i = FALSE

40. Binary search (Ada)
©Ian Sommerville 1995

41. Binary search (C++) void Binary_search (elem key, elem* T, int size,
boolean &found, int &L) {
int bott, top, mid ;
bott = 0 ; top = size -1 ;
L = ( top + bott ) / 2 ;
if (T[L] == key)
found = true ;
else
found = false ;
while (bott <=top && !found)
mid = top + bott / 2 ;
if ( T [mid] == key )
found = true;
L = mid ; }
else if (T [mid] < key )
bott = mid + 1 ;
top = mid-1 ;
} // while
} //binary_search
©Ian Sommerville 1995

42. Binary search - equiv. partitions
Pre-conditions satisfied, key element in array
Pre-conditions satisfied, key element not in array
Pre-conditions unsatisfied, key element in array
Pre-conditions unsatisfied, key element not in array
Input array has a single value
Input array has an even number of values
Input array has an odd number of values
©Ian Sommerville 1995

43. Binary search equiv. partitions
©Ian Sommerville 1995

44. Binary search - test cases
©Ian Sommerville 1995

45. Code Coverage Statement coverage
Elementary statements: assignment, I/O, call
Select a test set T such that by executing P for each case in T, each statement of P is executed at least once.
Read(x); read(y);
if x > 0 then write(“1”);
else write(“2”);
if y > 0 then write(“3”);
else write(“4”);
T: {<x = -13, y = 51>, <x = 2, y = -3>}

46. White-box Testing: Determining the Paths
FindMean (FILE ScoreFile)
{ float SumOfScores = 0.0;
int NumberOfScores = 0;
float Mean=0.0; float Score;
Read(ScoreFile, Score);
while (! EOF(ScoreFile) {
if (Score > 0.0 ) {
SumOfScores = SumOfScores + Score;
NumberOfScores++; }
/* Compute the mean and print the result */
if (NumberOfScores > 0) {
Mean = SumOfScores / NumberOfScores;
printf(“ The mean score is %f\n”, Mean);
} else
printf (“No scores found in file\n”); 1 2 3 4 5 6 7 8 9

47. Constructing the Logic Flow Diagram

48. Code Coverage - Construct a control-flow graph of a program (module) c
Assignment,
I/O, call
If c then S1
else S2
If c then S1
while c S1
Edge coverage
Select a set T such that, by executing P for each member in T,
each edge of P’s control-flow graph is traversed at least
once.

49. Code Coverage Condition coverage Edge coverage plus
All possible values of the constituents of compound conditions are exercised at least once.
(constituents: atomic formula  rel./Boolean var)
Example:
while (~found) and counter <= NumOfItems

50. Code Coverage Path coverage
Path coverage focuses on executing distinct paths rather than just edges or statements
Paths through a program can be feasible or infeasible (e.g., due to contradicting conditions)
Ideally, one would like to cover all feasible paths, but the number of feasible paths usually explodes very quickly with the size of the program
Therefore, there are various path coverage strategies targeting specific subsets of the feasible paths

51. Code Coverage Path coverage
- Select a test set T such that, by executing P for each member of
T, all paths leading from the initial node to the final node of P’s
control-flow graph are traversed.
- Control-flow graph S1 S c
Seq.
do S until c
If c
While c then S S2
If c then S1 else S2
Note: Separate node for each member of a compound condition
How many paths?

52. Code Coverage Path testing based on cyclomatic complexity
(McCabe’s) Cyclomatic complexity
V(G) = E – N + 2  E: # of edges, N: # of nodes
V(G) = p + 1  p is the # of predicate nodes
V(G) = # of regions (area surrounded by nodes/edges)
V(G): Upper bound on the # of independent paths
Independent path: A path with at least one new node/edge
Example:
V(G) = E – N + 2 = = 6
V(G) = p + 1 = 5 + 1
V(G) = 6
Advantage: The number of test cases is proportional to the size of the program

53. Code Coverage Path coverage: example 1 : 2 1 F 3 2 3 4 R1 5 6 F 4 7 R4
I = 1;
TI = TV = 0;
sum = 0;
DO WHILE (value[I] <> -999 and TI < 100)
TI++;
if (value[I] >= min and value[I] <= max){
then {TV++; sum = sum + value[I];} }
I++;
ENDDO
If TV > 0
av = sum/TV;
Else av = -999; 2 1 F 3 2 3 4 R1 5 6 F 4 7 R4 5 8 F 10 R2 T 9 6 T R5 10 12 11 R3 11 8 7 13 12 R6 9
13  final node
Outer region

54. V(G) as An Upper Bound 1 2 3 1 2
V(G) = 3

55. V(G)-Preserving Transformation
SWITCH
V(G) = 3
Nested IFs
V(G) = 3

56. Code Coverage
More complete path testing strategies target loops, e.g.,
0 and 1 passes through every loop is tested
Multiple passes exercising different paths through the body of a loop
n passes (max = n)
Trying to be more complete on loop testing quickly becomes infeasible

57. Code Coverage Data flow testing ( D = Definition, U = Use) DU chain:
Assign a unique number to each statement
DEF(S): set of all variables defined in statement number S
USE(S): set of all variables used in statement number S
Live variable: Def.of var X at S is live at S’ if
There is a path from S to S’ that contains no other def of X
DU chain: [X, S, S’]
X is in DEF(S) and in USE(S’), and
Def of X at S is live at S’
DU coverage
Every DU chain be covered at least once.

58. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

59. Unit Testing
Objective: Find differences between specified units and their imps.
Unit: component ( module, function, class, objects, …)
Unit test environment:
Driver
Test cases
Test result
Effectiveness?
Partitioning
Code coverage
Unit under
test
Stub
Stub
Dummy modules

60. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

61. Integration Testing Objectives: Problem areas
To expose problems arising from the combination
To quickly obtain a working solution from comps.
Problem areas
Internal: between components
Invocation: call/message passing/…
Parameters: type, number, order, value
Invocation return: identity (who?), type, sequence
External:
Interrupts (wrong handler?)
I/O timing
Interaction

62. Integration Testing Types of integration Structural Behavioral
“Big bang”  no error localization
Bottom-up: terminal, driver/module, (driver  module)
Top-down: top, stubs, (stub  module), early demo
Behavioral
(next slide)

63. Integration Testing (Behavioral: Path-Based)C
MM-path: Interleaved sequence of module exec path and messages
Module exec path: entry-exit path in the same module
Atomic System Function: port input, … {MM-paths}, … port output
Test cases: exercise ASFs

64. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

65. System Testing Concerns with the app’s externals
Much more than functional
Load/stress testing
Usability testing
Performance testing
Resource testing

66. System Testing Functional testing
Objective: Assess whether the app does what it is supposed to do
Basis: Behavioral/functional specification
Test case: A sequence of ASFs (thread)
(Refer to pages of ECE 355 PBX Project Desc.)

67. System Testing Functional testing: coverage Event-based coverage
PI1: each port input event occurs
PI2: common sequences of port input event occurs
PI3: each port input in every relevant data context
PI4: for a given context, all possible input events
PO1: each port output event
PO2: each port output event occurs for each cause
Data-based
DM1: Exercise cardinality of every relationship
DM2: Exercise (functional) dependencies among relationships

68. System Testing Stress testing: push it to its limit + beyond Volume
Application
(System)
Users
response :
rate
Resources: phy. + logical

69. System Testing Performance testing Usability testing
Performance seen by
users: delay, throughput
System owner: memory, CPU, comm
Performance
Explicitly specified or expected to do well
Unspecified  find the limit
Usability testing
Human element in system operation
GUI, messages, reports, …

70. Test Stopping Criteria
Meet deadline, exhaust budget, …  management
Achieved desired coverage
Achieved desired level failure intensity

71. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

72. Acceptance Testing Purpose: ensure that end users are satisfied
Basis: user expectations (documented or not)
Environment: real
Performed: for and by end users (commissioned projects)
Test cases:
May reuse from system test
Designed by end users

73. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

74. Regression Testing
Whenever a system is modified (fixing a bug, adding functionality, etc.), the entire test suite needs to be rerun
Make sure that features that already worked are not affected by the change
Automatic re-testing before checking in changes into a code repository
Incremental testing strategies for big systems

75. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations

76. How to Design Practical Test Cases (Experience of Hitachi Software Engineering)
Bug detection during a development phase
100%
50% 0%
Desk
Unit
Integrate
System
Field failure
0.02%
10 engineers x 12 months  100,000 LOC
1000 bugs  < 1 bug appears at the customer’s site

77. Testing effectiveness
In an experiment, black-box testing was found to be more effective than structural testing in discovering defects
Static code reviewing was less expensive and more effective in discovering program faults

78. Practical test cases Document test cases
New viewpoint to functional spec
Validation of test cases
Well-balanced: normal, abnormal, boundary
Correctness test cases
Estimating quality
100/1000 test cases show 4 bugs  40 bugs

79. Practical test cases Schedule (rough idea)
10 engineers, 12 months, 100,000 LOC in C
Apportioning 12 months
SRS: 2; Design: 3; Coding: 2
Debugging (U + I + S tests): 3
QA testing: 2 (shippable??, redesign tests—no reuse)
How many test cases (30-year empirical study)
1 test case per LOC
100,000 LOC  1000 test cases
2 weeks to design 1000 test cases
2 months to execute 1000 test cases (25/day)

80. Comparison of White & Black-box Testing 25.1.2002
White-box Testing:
Potentially infinite number of paths have to be tested
White-box testing often tests what is done, instead of what should be done
Cannot detect missing use cases
Black-box Testing:
Potential combinatorical explosion of test cases (valid & invalid data)
Often not clear whether the selected test cases uncover a particular error
Does not discover extraneous use cases ("features")
Both types of testing are needed
White-box testing and black box testing are the extreme ends of a testing continuum.
Any choice of test case lies in between and depends on the following:
Number of possible logical paths
Nature of input data
Amount of computation
Complexity of algorithms and data structures

81. The 4 Testing Steps 1. Select what has to be measured
Analysis: Completeness of requirements
Design: tested for cohesion
Implementation: Code tests
2. Decide how the testing is done
Code inspection
Proofs (Design by Contract)
Black-box, white box,
Select integration testing strategy (big bang, bottom up, top down, sandwich)
3. Develop test cases
A test case is a set of test data or situations that will be used to exercise the unit (code, module, system) being tested or about the attribute being measured
4. Create the test oracle
An oracle contains of the predicted results for a set of test cases
The test oracle has to be written down before the actual testing takes place

82. Guidance for Test Case Selection
Use analysis knowledge about functional requirements (black-box testing):
Use cases
Expected input data
Invalid input data
Use design knowledge about system structure, algorithms, data structures (white-box testing):
Control structures
Test branches, loops, ...
Data structures
Test records fields, arrays, ...
Use implementation knowledge about algorithms:
Examples:
Force division by zero
Use sequence of test cases for interrupt handler

83. Unit-testing Heuristics
1. Create unit tests as soon as object design is completed:
Black-box test: Test the use cases & functional model
White-box test: Test the dynamic model
Data-structure test: Test the object model
2. Develop the test cases
Goal: Find the minimal number of test cases to cover as many paths as possible
3. Cross-check the test cases to eliminate duplicates
Don't waste your time!
4. Desk check your source code
Reduces testing time
5. Create a test harness
Test drivers and test stubs are needed for integration testing
6. Describe the test oracle
Often the result of the first successfully executed test
7. Execute the test cases
Don’t forget regression testing
Re-execute test cases every time a change is made.
8. Compare the results of the test with the test oracle
Automate as much as possible

84. Practical test cases Designing test cases (contd.)
Distribution of test cases
Normal: 60%
Boundary: 10%
Error: 15%
Environmental (platform + performance): 15%
Finishing touch
48-hour continuous operation test (basic function)
 memory leakage, deadlock, connection time-out

85. Overview Basics of Testing Testing & Debugging Activities
Testing Strategies
Black-Box Testing
White-Box Testing
Testing in the Development Process
Unit Test
Integration Test
System Test
Acceptance Test
Regression Test
Practical Considerations