Software Reliability

Risks of faulty software  Example: –Therak 25, –AT&T network failure –Airport traffic control  Costs of software errors can be huge –Y2K problem detection, fix, litigation

Testing  Testing is the process of searching for errors, with the goal of finding as many as possible so that they can be turned over to those responsible for fixing them

Types of errors  Syntax errors - typos, easy to fix  Validity errors - wrong result  Verification errors - undetected bad input  Run-time errors - General Protection Fault  Maintenance errors –one fix produces another problem  Cost to repair errors –tied to ease of identification

Two approaches to testing  Black-box testing –Only inputs and outputs are tested –Code is ignored (black box)  Glass-box testing –Testing based on enumerated special cases and anticipated possible inputs –Must examine code to see what conditions it expects to see

Unit vs. System Testing  Design->implementation-> unit testing-> system testing  Unit testing –Testing individual components of the system –Easy to isolate bugs this way –Based on the program design  System testing –After units check out do a ‘system build’ –Based on the program specifications

Unit testing  Unit testing handles the individual components one at a time.  In OOD the units to be tested are the various classes  It is important to catch bugs at this level, rather than later on after the system has been integrated.

System integration  After unit testing is finished you are ready for a ‘system build’  You are looking for errors in the overall operation of the system.  Follow the specifications that preceded the design.

Acceptance testing  This form of system testing is overseen by the client to insure that the program meets their specs.  Functional testing - testing to demonstrate that the function of each specification is met.

System testing components  Look at specifications for the end product  Some errors may require redesign  Functional testing –Specifying required functions which must work  Beta testing –Software release to ‘guinea pigs’

Test plans  Used by trained testing teams  Scientific method –(hypothesis verification)  Test cases – 1. Typical ones – 2. Extreme cases – 3. Invalid input

Example  Specifications: –Write a program called Max that will read in a sequence of integers and print out the largest one entered.  A test plan for Max should handle all types of errors.

Test plan for Max program

Unit testing techniques  Drivers –Small programs designed to test a function –A driver program calls the function with a set of test data, and checks to see if the function processes it correctly. –A suitable value is returned by the function, but the driver program really does not need it

Test driver for max function (Part 1 of 2)  #include  int max(int a[], int n); // defined in cx3-3.cpp; link with the driver  int main()  { int a[100], i;  cout << "Max driver” << endl;  cout << "Enter numbers terminated by -9999”;  cout << endl;  cout << "Length of input must be <= 100”;  cout << endl;

Test driver for max function (Part 2 of 2) for (i = 0; i < 100; i++) { int val; cin >> val; if (val == -9999) // sentinel break; else a[i] = val; } cout << "\nMax is " << max(a, i) << endl; cout << endl << endl; return 0; }

Stubs  A stub is –a function that does nothing more than give calling routines what they ask for. –Used to test a function that depends on other functions that are not yet available.

Sample stub for function median  #include  int median(int a[], int n)  {cout << "function median called with n = " << n <<;  cout << “a[] = ";  int i;  for (i = 0; i < n; i++)  cout << a[i] << '\t';  cout << "\nType in a value to return:";  int return_value;  cin >> return_value;  return return_value;  }

Alternatives to testing  The problem with testing is that you cannot test all possible situations.  So you can never ‘prove’ that the program works.  A better way of testing would be using a mathematical proof of the programs’ correctness

Problems with proofs  However, proving that a program will perform correctly is not the same as a guarantee that it will do what you want it to.  We can show that the program specifications are being met, but we cannot show that a specification itself is correct.

Example  We could show that a vending machine program meets the spec. of dispensing the fewest number of coins possible as change.  We cannot show that this specification is the correct strategy (I.e. postal machines that give you Susan B. Anthony dollars)

Program Correctness Techniques  Proving program correctness is a primary goal of computer science, but a very broad topic  There is one aspect we can use.  Assertions –Precise statements of the behavior of the program at a particular spot.

Assertions  An assertion is a statement of the specifications of a contract.  The program can be evaluated to see if it lives up to the terms of the contract at any given point.  There are many key types of assertions

Assertions  Preconditions –Statements about what we expect to be true before the function begins  Loop invariants –Statements about conditions that are true for each iteration of the loop  Postconditions –What we expect the result of the function to be

Proof of correctness  A proof of correctness uses the loop invariant to show that –given a certain set of preconditions –the invariant guarantees that the postcondition will always be met.

Pre and post conditions  These are special assertions that describe how the program performs.  “If when function f is called A is true, then, when function f returns, B will be true.”

Example  If when the max function is called, it is true that ‘a’ is an array ranging from 0 to n-1, then, when max returns, the value returned will be equal to the largest value in ‘a’.

Contracts  Preconditions and postconditions are contracts between client code and the function.  If the preconditions are met by the client, then the function guarantees the specified result upon its return.

Broken contracts  The program has a bug in it if it cannot fulfill its contracts  One common way of breaking a contract is for client code to fail to meet the precondition  This happens whenever there is failure to correctly exchange data through the function interface.

Loop invariants  Loop invariants are statements that are true at the beginning and ending of each iteration of the loop.  Loops are another major source of broken contracts –they do not terminate at the right time –they do not terminate at all –they do not execute

Code Example Function max  int max (int a[], int n)  { // assertion 1: THE PRECONDITION // a is an array with subscripts from 0 to n-1  int max_val(a[0]), i;  for (i = 1; i < n; i++)  // assertion 2: THE LOOP INVARIANT // (max_val >= a[k] for 0 <= k < i) and  // (max_val = a[j] for some j, 0 <= j < i)  if (max_val < a[i]) max_val = a[i];  // assertion 3: THE POST CONDITION // (max_val >= a[k] for 0 <= k < n) and  // (max_val == a[j] for some j,0 <= j< n)  // i.e., max_val is the value of largest int in a  return max_val;  }

Assertions are like contracts  Guarantees between program segments  Function interfaces are prime spots for bugs –General Protection Faults (Turbo) –Segmentation faults (unix)

Assert.h  #include  int max (int a[], int); // See next slide  int main()  {  int a[] = { 45,21,76,34,62,58,92,34,10,98 };  int n = 10;  cout << "The largest is: " << max(a,n) << endl;  return 0;  }

Example: assertions  int max (int a[], int n)  { // assertion 1: a is an array with subscripts ranging from 0 to n-1  assert (0 < n);// n is positive  int max_val(a[0]), i;  for (i = 1; i < n; i++) {  if (max_val < a[i]) max_val = a[i];  // assertion 2: (max_val >= a[k] for 0 <= k < i) and  // (max_val = a[j] for some j, 0 <= j < i)  assert (max_val >= a[i]); }  // assertion 3: (max_val >= a[k] for 0 <= k < n) and  // (max_val == a[j] for some j, 0 <= j < n)  // i.e., max_val is equal the value of largest int in array a  return max_val; }

Loop invariants  Anything that is true for every iteration is a loop invariant  There are many possible invariants  i == i is true but uninteresting  i < n is also true but uninteresting, even though it is the termination condition

Important invariants  Operant condition –(max_val >= a[k] for 0 <= k < i) and –(max_val == a[j] for some j, 0 <= j < i)  After the loop is over we can substitute n for the value of i –(max_val >= a[k] for 0 <= k < n) and –(max_val == a[j] for some j, 0 <= j < n)  Now we have the post condition!

Establishing a loop invariant  We must show that two things are true –The invariant is true the first time the loop is entered. –If it is true for one interation it is true for the next as well.

The invariant is true the first time  When we enter the loop i is 1, therefore we have: –(max_val >= a[k] for 0 <= k < 1) and –(max_val == a[j] for some j, 0 <= j < 1)  This is true since k and j are 0

The invariant is true for each successive time  For all i, a[i] is either greater that max_val or not  If it is, max_val becomes a[i]  If it is not, max_val stays unaltered

Illustrated loop invariant for function max max_val >= everything in here this part is unknown i

Proving termination  How do we know the loop will end?  Because i starts out less than n and is incremented in each iteration - it will eventually get to n no matter what n is.

Steps for analyzing a loop  Establish the precondition  Find the invariant  Establish the postcondition  Prove termination

sum function  int sum(int a[], int n)  { // Precondition: a is an array with // subscripts from 0 to n-1  int i; total(0);  for (i = 0; i < n; i++)  // Loop invariant: // total = a[0] + a[1] + … + a[i]  total += a[i];  // Postcondition: // total = a[0] + a[1] + … + a[n-1]  return total;  }

Example  The insertion sort

Basic idea of Insertion Sort Unprocessed Items

Insertion Sort  void insertNextItem(int a[], int i);  void insertionSort(int a[], int n)  {// Precondition: // a is an array with subscripts from 0 to n-1  for (int i = 1; i < n; i++)  // Loop invariant: // items in range from 0 to i-1 are sorted;  // items from i to n-1 haven’t been examined.  insertNextItem(a, i);  // Postcondition: array a is sorted  }

InsertNextItem  void insertNextItem(int a[], int i)  {// Precondition: array a is sorted from 0 to i-1  int newItem(a[i]), insertPos(i);  for (;insertPos && newItem < a[insertPos-1];insertPos--)  // Loop Invariant: newItem <= a[insertPos+1].. a[i] &&  // a[insertPos+1].. a[i] are sorted  a[insertPos] = a[insertPos-1]; // slide item right  a[insertPos] = newItem;  // Postcondition: array a is sorted from 0 to i  }

Proof of termination  Two ways to exit the loop –insertPos <= 0 –newItem < a[insertPos - 1]  We must show that one of these eventually happens  If insertPos starts out greater than 0 then the first will always be met since we subtract 1 each iteration.