Chapter 26 Testing Bjarne Stroustrup www.stroustrup.com/Programming.

Slides:



Advertisements
Similar presentations
Chapter 10 Input/Output Streams
Advertisements

Chapter 17 vector and Free Store Bjarne Stroustrup
Chapter 3 Objects, types, and values Bjarne Stroustrup
Chapter 5 Errors Bjarne Stroustrup
Chapter 12 A display model Bjarne Stroustrup
Chapter 8 Technicalities: Functions, etc. Bjarne Stroustrup
Chapter 7 Completing a Program
Review for Final Exam Chapters 1–20 (Programming and C++ in 75 minutes) Bjarne Stroustrup Ronnie Ward 1Stroustrup/PPP -
Chapter 24 Numerics Bjarne Stroustrup
Chapter 4 Computation Bjarne Stroustrup
Chapter 6 Writing a Program
By Rick Clements Software Testing 101 By Rick Clements
Software Testing with QuickCheck Lecture 1 Properties and Generators.
Chapter 10 Software Testing
Chapter 17 vector and Free Store John Keyser’s Modifications of Slides By Bjarne Stroustrup
Automatic Verification Book: Chapter 6. What is verification? Traditionally, verification means proof of correctness automatic: model checking deductive:
Testing and Quality Assurance
Annoucements  Next labs 9 and 10 are paired for everyone. So don’t miss the lab.  There is a review session for the quiz on Monday, November 4, at 8:00.
1 Basic Definitions: Testing What is software testing? Running a program In order to find faults a.k.a. defects a.k.a. errors a.k.a. flaws a.k.a. faults.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 23 Slide 1 Software testing.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 23 Slide 1 Software testing.
OOP #10: Correctness Fritz Henglein. Wrap-up: Types A type is a collection of objects with common behavior (operations and properties). (Abstract) types.
Finding the Weakest Characterization of Erroneous Inputs Dzintars Avots and Benjamin Livshits.
1 Today More on random testing + symbolic constraint solving (“concolic” testing) Using summaries to explore fewer paths (SMART) While preserving level.
Lecture , 3.1 Methods of Proof. Last time in 1.5 To prove theorems we use rules of inference such as: p, p  q, therefore, q NOT q, p  q, therefore.
Introduction To C++ Programming 1.0 Basic C++ Program Structure 2.0 Program Control 3.0 Array And Structures 4.0 Function 5.0 Pointer 6.0 Secure Programming.
1 Functional Testing Motivation Example Basic Methods Timing: 30 minutes.
Quadratic Residuosity and Two Distinct Prime Factor ZK Protocols By Stephen Hall.
28/08/2015SJF L31 F21SF Software Engineering Foundations ASSUMPTIONS AND TESTING Monica Farrow EM G30 Material available on Vision.
Dr. Pedro Mejia Alvarez Software Testing Slide 1 Software Testing: Building Test Cases.
Lecture 18 Page 1 CS 111 Online Design Principles for Secure Systems Economy Complete mediation Open design Separation of privileges Least privilege Least.
Software Quality Assurance Lecture #8 By: Faraz Ahmed.
1 Debugging and Testing Overview Defensive Programming The goal is to prevent failures Debugging The goal is to find cause of failures and fix it Testing.
Chapter 8 – Software Testing Lecture 1 1Chapter 8 Software testing The bearing of a child takes nine months, no matter how many women are assigned. Many.
Computer Programming I An Introduction to the art and science of programming with C++
CSE 219 Computer Science III Testing. Testing vs. Debugging Testing: Create and use scenarios which reveal incorrect behaviors –Design of test cases:
1 Today Random testing again Some background (Hamlet) Why not always use random testing? More Dominion & project? CUTE: “concolic” testing.
Chapter 13 Recursion. Learning Objectives Recursive void Functions – Tracing recursive calls – Infinite recursion, overflows Recursive Functions that.
Testing and Debugging Version 1.0. All kinds of things can go wrong when you are developing a program. The compiler discovers syntax errors in your code.
Testing. 2 Overview Testing and debugging are important activities in software development. Techniques and tools are introduced. Material borrowed here.
Introduction to Software Testing. Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing.
Unit Testing 101 Black Box v. White Box. Definition of V&V Verification - is the product correct Validation - is it the correct product.
How Responsible and Respectful are you in the Computer Lab Created for the Students At Donaldson By Mrs. Jones.
Mannheim Research Institute for the Economics of Aging SHARE Data Cleaning Stephanie Stuck MEA Vienna November 5/6 th.
What is Testing? Testing is the process of finding errors in the system implementation. –The intent of testing is to find problems with the system.
Chapter 8 Lecture 1 Software Testing. Program testing Testing is intended to show that a program does what it is intended to do and to discover program.
Halting Problem and TSP Wednesday, Week 8. Background - Halting Problem Common error: Program goes into an infinite loop. Wouldn’t it be nice to have.
Digital Logic Lecture 3 Binary Arithmetic By Zyad Dwekat The Hashemite University Computer Engineering Department.
CSE 374 Programming Concepts & Tools Hal Perkins Fall 2015 Lecture 11 – gdb and Debugging.
Dale Roberts Department of Computer and Information Science, School of Science, IUPUI Dale Roberts, Lecturer Computer Science, IUPUI
ANU COMP2110 Software Design in 2003 Lecture 10Slide 1 COMP2110 Software Design in 2004 Lecture 12 Documenting Detailed Design How to write down detailed.
CSCI1600: Embedded and Real Time Software Lecture 28: Verification I Steven Reiss, Fall 2015.
Understanding ADTs CSE 331 University of Washington.
Random Numbers Random numbers are extremely useful: especially for games, and also for calculating experimental probabilities. Formula for generating random.
PROGRAMMING TESTING B MODULE 2: SOFTWARE SYSTEMS 22 NOVEMBER 2013.
Refactoring1 Improving the structure of existing code.
PROGRAMMING PRE- AND POSTCONDITIONS, INVARIANTS AND METHOD CONTRACTS B MODULE 2: SOFTWARE SYSTEMS 13 NOVEMBER 2013.
CS116 COMPILER ERRORS George Koutsogiannakis 1. How to work with compiler Errors The Compiler provide error messages to help you debug your code. The.
Loop Invariants and Binary Search Chapter 4.4, 5.1.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 23 Slide 1 Software testing.
Overview of the Operations Research Modeling Approach Chapter 2: Hillier and Lieberman Chapter 2: Decision Tools for Agribusiness Dr. Hurley’s AGB 328.
Defensive Programming. Good programming practices that protect you from your own programming mistakes, as well as those of others – Assertions – Parameter.
Types for Programs and Proofs
CSE 374 Programming Concepts & Tools
Lessons from The File Copy Assignment
Chapter 8 – Software Testing
Boolean Algebra Introduction CSCI 240
Test Case Test case Describes an input Description and an expected output Description. Test case ID Section 1: Before execution Section 2: After execution.
  Pick a card….
Computer Science 340 Software Design & Testing
Presentation transcript:

Chapter 26 Testing Bjarne Stroustrup

Abstract This lecture is an introduction to the design and testing of program units (such as functions and classes) for correctness. We discuss the use of interfaces and the selection of tests to run against them. We emphasize the importance of designing systems to simplify testing and testing from the start. Proving programs correct and performance problems are also briefly considered. This lecture is an introduction to the design and testing of program units (such as functions and classes) for correctness. We discuss the use of interfaces and the selection of tests to run against them. We emphasize the importance of designing systems to simplify testing and testing from the start. Proving programs correct and performance problems are also briefly considered. 2Stroustrup/Programming

Overview Correctness, proofs, and testing Correctness, proofs, and testing Dependencies Dependencies System tests System tests Testing GUIs Testing GUIs Resource management Resource management Unit and system tests Unit and system tests Finding assumptions that do not hold Finding assumptions that do not hold Design for testing Design for testing Performance Performance 3Stroustrup/Programming

Correctness Questions to ask about a program Questions to ask about a program Is your program correct? Is your program correct? What makes you think so? What makes you think so? How sure are you? How sure are you? Why? Why? Would you fly in a plane that depended on that code? Would you fly in a plane that depended on that code? You have to be able to reason about your code to have any real certainty You have to be able to reason about your code to have any real certainty Programming is generally unsystematic Programming is generally unsystematic Debugging is generally unsystematic Debugging is generally unsystematic What are you willing to bet that you found the last bug? What are you willing to bet that you found the last bug? Related interesting questions Related interesting questions Could the program run forever if the hardware didnt fail? Could the program run forever if the hardware didnt fail? Does it always deliver its results in a reasonable time? Does it always deliver its results in a reasonable time? 4Stroustrup/Programming

Proofs So why not just prove mathematically that our program is correct? So why not just prove mathematically that our program is correct? Its often too hard and/or takes too long Its often too hard and/or takes too long Sometimes proofs are wrong too (even proofs produced by computers or by experts!). Sometimes proofs are wrong too (even proofs produced by computers or by experts!). Computer arithmetic isnt the same as real mathremember the rounding and overflow errors we saw (due to finite and limited precision)? Computer arithmetic isnt the same as real mathremember the rounding and overflow errors we saw (due to finite and limited precision)? So we do what we can: follow good design principles, test, test, and then test some more! So we do what we can: follow good design principles, test, test, and then test some more! 5Stroustrup/Programming

Testing A systematic way to search for errors A systematic way to search for errors Real testers use a lot of tools Real testers use a lot of tools Unit test frameworks Unit test frameworks Static code analysis tools Static code analysis tools Fault injection tools Fault injection tools … When done well, testing is a highly skilled and most valuable activity When done well, testing is a highly skilled and most valuable activity Test early and often Test early and often Whenever you write a function or a class, think of how you might test it Whenever you write a function or a class, think of how you might test it Whenever you make a significant change, re-test Whenever you make a significant change, re-test Before you ship (even after the most minor change), re-test Before you ship (even after the most minor change), re-test 6Stroustrup/Programming

Testing Some useful sets of values to check (especially boundary cases): Some useful sets of values to check (especially boundary cases): the empty set the empty set small sets small sets large sets large sets sets with extreme distributions sets with extreme distributions sets where what is of interest happens near the ends sets where what is of interest happens near the ends sets with duplicate elements sets with duplicate elements sets with even and with odd number of elements sets with even and with odd number of elements some sets generated using random numbers some sets generated using random numbers 7Stroustrup/Programming

Primitive test harness for binary_search() int a1[ ] = { 1,2,3,5,8,13,21 }; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),1) == false) cout << "1 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),5) == false) cout << "2 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),8) == false) cout << "3 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),21) == false) cout << "4 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),-7) == true) cout << "5 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),4) == true) cout << "6 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),22) == true) cout << "7 failed"; 8Stroustrup/Programming

A Better Test Harness (stil primitive) Put the variables into a data file, e.g., with a format of { 27 7 { } 0 } meaning {test_number value {sequence} result} i.e., test #27 calls our binary_search to look for the value 7 in the sequence { } and checks that the result is 0 (false, that is, not found). Now its (relatively) easy to write lots of test cases, or even write another program to generate a data file with lots of (random) cases. 9Stroustrup/Programming

Dependencies Basically we want every function to: have well-defined inputs have well-defined inputs have well-defined results have well-defined results including any modifications to input parameters including any modifications to input parameters in a determinate amount of time (no infinite loops, please) in a determinate amount of time (no infinite loops, please) not have dependencies on objects that are not its explicit inputs not have dependencies on objects that are not its explicit inputs Hard to achieve in real life Hard to achieve in real life not use more resources than are available and appropriate not use more resources than are available and appropriate E.g., time, memory, internet bandwidth, files, and locks E.g., time, memory, internet bandwidth, files, and locks 10Stroustrup/Programming

Dependencies How many dependencies can you spot in this nonsense function? int do_dependent(int a, int& b) // messy function // undisciplined dependencies // undisciplined dependencies{ int val ; cin>>val; vec[val] += 10; cout << a; b++; return b; } 11Stroustrup/Programming

Resource Management What resources (memory, files, etc.) acquired may not always be properly released in this nonsense function? void do_resources1(int a, int b, const char* s) // messy function // undisciplined resource use // undisciplined resource use{ FILE* f = fopen(s,"r"); // open file (C style) int* p = new int[a]; // allocate some memory if (b<=0) throw Bad_arg(); // maybe throw an exception int* q = new int[b]; // allocate some more memory delete[ ] p; // deallocate the memory pointed to by p } 12Stroustrup/Programming

Better Resource Management // less messy function void do_resources2(int a, int b, const string& s) { istream is(s.c_str(),"r"); // open file vector v1(a); // create vector (owning memory) if (b<=0) throw Bad_arg(); // maybe throw an exception vector v2(b); // create another vector (owning memory) } Can do_resources2() leak anything? 13Stroustrup/Programming

Loops Most errors occur at the ends, i.e., at the first case or the last case. Can you spot 3 problems in this code? 4? 5? int do_loop(vector & vec) // messy function // undisciplined loop // undisciplined loop{ int i; int sum; while(i<=vec.size()) sum+=v[i]; return sum; } 14Stroustrup/Programming

Buffer Overflow Really a special type of loop error, e.g., storing more bytes than will fit into an arraywhere do the extra bytes go? (probably not a good place) Really a special type of loop error, e.g., storing more bytes than will fit into an arraywhere do the extra bytes go? (probably not a good place) The premiere tool of virus writers and crackers (evil hackers) The premiere tool of virus writers and crackers (evil hackers) Some vulnerable functions (best avoided): Some vulnerable functions (best avoided): gets, scanf// these are the worst: avoid! gets, scanf// these are the worst: avoid! sprintf sprintf strcat strcat strcpy strcpy … 15Stroustrup/Programming

Buffer overflow Dont avoid unsafe functions just as a fetish Dont avoid unsafe functions just as a fetish Understand what can go wrong and dont just write equivalent code Understand what can go wrong and dont just write equivalent code Even unsafe functions (e.g. strcpy()) have uses Even unsafe functions (e.g. strcpy()) have uses if you really want to copy a zero terminated string, you cant do better than strcpy() – just be sure about your strings (How?) if you really want to copy a zero terminated string, you cant do better than strcpy() – just be sure about your strings (How?) char buf[MAX]; char* read_line()// harmless? Mostly harmless? Avoid like the plague? { int i = 0; char ch; while (cin.get(ch) && ch!='\n') buf(i++)=ch; buf[i+1]=0; return buf; } Stroustrup/Programming16

Buffer overflow Dont avoid unsafe functions just as a fetish Dont avoid unsafe functions just as a fetish Understand what can go wrong and dont just write equivalent code Understand what can go wrong and dont just write equivalent code Write simple and safe code Write simple and safe code string buf; getline(cin,buf); // buf expands to hold the newline terminated input Stroustrup/Programming17

Branching In if and switch statements In if and switch statements Are all alternatives covered? Are all alternatives covered? Are the right actions associated with the right conditions? Are the right actions associated with the right conditions? Be careful with nested if and switch statements Be careful with nested if and switch statements The compiler ignores your indentation The compiler ignores your indentation Each time you nest you must deal with all possible alternatives Each time you nest you must deal with all possible alternatives Each level multiplies the number of alternatives (not just add) Each level multiplies the number of alternatives (not just add) For switch statements For switch statements remember the default case and to break after each other case remember the default case and to break after each other case unless you really meant to fall through unless you really meant to fall through 18Stroustrup/Programming

System Tests Do unit tests first, then combinations of units, and so on, till we get to the whole system Do unit tests first, then combinations of units, and so on, till we get to the whole system Ideally, in isolation from other parts of the system Ideally, in isolation from other parts of the system Ideally, in a repeatable fashion Ideally, in a repeatable fashion What about testing GUI based applications? What about testing GUI based applications? Control inversion makes GUI testing difficult Control inversion makes GUI testing difficult Human behavior is not exactly repeatable Human behavior is not exactly repeatable Timing, forgetfulness, boredom, etc. Timing, forgetfulness, boredom, etc. Humans still needed at some point (only a human can evaluate look and feel) Humans still needed at some point (only a human can evaluate look and feel) Simulate user input from a test script Simulate user input from a test script That way a test harness script takes the place of the human for many tests That way a test harness script takes the place of the human for many tests An excellent application of layering with well-defined interfaces between the layers An excellent application of layering with well-defined interfaces between the layers Allows for portability of applications across GUI systems Allows for portability of applications across GUI systems A GUI is often used as a lock-in mechanism A GUI is often used as a lock-in mechanism 19Stroustrup/Programming

Testing Classes A type of unit test A type of unit test but most class objects have state but most class objects have state Classes often depend on interactions among member functions Classes often depend on interactions among member functions A base class must be tested in combination with its derived classes A base class must be tested in combination with its derived classes Virtual functions Virtual functions Construction/initialization is the combined responsibility of several classes Construction/initialization is the combined responsibility of several classes Private data is really useful here (beware of protected data members) Private data is really useful here (beware of protected data members) Take our Shape class as an example: Take our Shape class as an example: Shape has several functions Shape has several functions A Shape has a mutable state (we can add points, change color, etc.); that is, the effect of one function can affect the behavior of another function A Shape has a mutable state (we can add points, change color, etc.); that is, the effect of one function can affect the behavior of another function Shape has virtual functions; that is, the behavior of a Shape depends on what (if any) class has been derived from it Shape has virtual functions; that is, the behavior of a Shape depends on what (if any) class has been derived from it Shape is not an algorithm (why not?) Shape is not an algorithm (why not?) A change to a Shape can have an effect on the screen (so maybe we still need a human tester?) A change to a Shape can have an effect on the screen (so maybe we still need a human tester?) 20Stroustrup/Programming

Finding assumptions that do not hold For example, illegal input arguments For example, illegal input arguments Should never happen, but it does Should never happen, but it does Check before each call or at the beginning of the function Check before each call or at the beginning of the function Depending on which code we can modify Depending on which code we can modify E.g., sqrt first checks that its argument is a non-negative value E.g., sqrt first checks that its argument is a non-negative value That can be difficult/problematic: That can be difficult/problematic: Consider binary_search(a,b,v); // is v in [a:b) Consider binary_search(a,b,v); // is v in [a:b) For forward iterators (e.g., for a list), we cant test if a<b – no < operation For forward iterators (e.g., for a list), we cant test if a<b – no < operation For random-access iterators,we cant check if a and b are part of the same sequence For random-access iterators,we cant check if a and b are part of the same sequence The only perfect solution involves a run-time checking library The only perfect solution involves a run-time checking library Scanning the entire sequence to verify its sorted is much more work than actually doing the binary search Scanning the entire sequence to verify its sorted is much more work than actually doing the binary search The purpose of binary_search() is to be faster than linear search The purpose of binary_search() is to be faster than linear search Sometimes, check in debug/test mode only Sometimes, check in debug/test mode only Leave (only) affordable tests in production code Leave (only) affordable tests in production code 21Stroustrup/Programming

Design for Testing Use well-defined interfaces Use well-defined interfaces so that you can write tests for the use of these interfaces so that you can write tests for the use of these interfaces Define invariants, pre- and post conditions Define invariants, pre- and post conditions Have a way of representing operations as text Have a way of representing operations as text so that they can be stored, analyzed and replayed so that they can be stored, analyzed and replayed Embed tests of unchecked assumptions (assertions) in the calling and/or called code Embed tests of unchecked assumptions (assertions) in the calling and/or called code to catch bad arguments before system testing to catch bad arguments before system testing Minimize dependencies and keep dependencies explicit Minimize dependencies and keep dependencies explicit To make it easier to reason about the code To make it easier to reason about the code Have a clear resource management strategy Have a clear resource management strategy This will also minimize debugging! 22Stroustrup/Programming

Performance Is it efficient enough? Is it efficient enough? Note: Not Is it as efficient as possible? Note: Not Is it as efficient as possible? Computers are fast: Youll have to do millions of operations to even notice (without using tools) Computers are fast: Youll have to do millions of operations to even notice (without using tools) Accessing permanent data (on disc) repeatedly can be noticed Accessing permanent data (on disc) repeatedly can be noticed Accessing the web repeatedly can be noticed Accessing the web repeatedly can be noticed Time interesting test cases Time interesting test cases e.g., using time or clock() e.g., using time or clock() Repeat 3 times; should be ± 10% to be believable Repeat 3 times; should be ± 10% to be believable 23Stroustrup/Programming

Performance Whats wrong with this? Whats wrong with this? for (int i=0; i<strlen(s); ++i) { // do something with s[i] } It was part of an internet message log analyzer It was part of an internet message log analyzer Used for files with many thousands of long log lines Used for files with many thousands of long log lines Stroustrup/Programming24

Using clock() int n = ; // repeat do_somenting() n times clock_t t1 = clock(); if (t1 = = clock_t(-1)) { // clock_t(-1) means clock() didn't work cerr << "sorry, no clock\n"; exit(1);} for (int i = 0; i<n; i++) do_something(); // timing loop clock_t t2 = clock(); if (t2 = = clock_t(-1)) { cerr << "sorry, clock overflow\n"; exit(2);} cout << "do_something() " << n << " times took " << double(t2-t1)/CLOCKS_PER_SEC << " seconds " // scale result << " (measurement granularity: " << CLOCKS_PER_SEC << " of a second)\n"; 25Stroustrup/Programming