Software testing Main issues: There are a great many testing techniques Often, only the final code is tested

SE, Testing, Hans van Vliet, ©2008 State-of-the-Art errors are made per 1000 lines of source code extensively tested software contains errors per 1000 lines of source code testing is postponed, as a consequence: the later an error is discovered, the more it costs to fix it. error distribution: 60% design, 40% implementation. 66% of the design errors are not discovered until the software has become operational.

SE, Testing, Hans van Vliet, ©2008 Relative cost of error correction REdesigncodetestoperation

SE, Testing, Hans van Vliet, ©2008 Lessons Many errors are made in the early phases These errors are discovered late Repairing those errors is costly ⇒ It pays off to start testing real early

SE, Testing, Hans van Vliet, ©2008 How then to proceed? Exhaustive testing most often is not feasible Proof correctness does not work for most cases Therefore, we look for systematic ways to proceed during testing

SE, Testing, Hans van Vliet, ©2008 Some preliminary questions What exactly is an error? How does the testing process look like? When is test technique A superior to test technique B? What do we want to achieve during testing? When to stop testing?

SE, Testing, Hans van Vliet, ©2008 Error, fault, failure an error is a human activity resulting in software containing a fault a fault is the manifestation of an error a fault may result in a failure

SE, Testing, Hans van Vliet, ©2008 When exactly is a failure a failure? Failure is a relative notion: e.g. a failure w.r.t. the specification document Verification: evaluate a product to see whether it satisfies the conditions specified at the start: Have we built the system right? Validation: evaluate a product to see whether it does what we think it should do: Have we built the right system?

SE, Testing, Hans van Vliet, ©2008 What is our goal during testing? Objective 1: find as many faults as possible Objective 2: make you feel confident that the software works OK

Unit Testing Goal: isolate smallest unit (using stubs, mocks, fakes) verify individual units are functioning correctly Smallest unit for functional programming: function or program for OO programming: method How? step through debugger use a framework (JUnit, etc.)

Unit Testing: Advantages find problems early allows change process: change code → run tests (build automation) → fix errors regression testing after refactoring integration becomes easy test parts, then the sum serves as documentation states class’s capability always up-to-date

Test-Driven Design (TDD) replace the formal design process start with writing tests implementation comes later used heavily in Agile or XP Development advantage over other design methods (UML) design document used for implementation, testing

Unit Testing: Limitations Not the best for integration testing can’t show the absence of errors (like others) test code may be buggy as well! requires lots of discipline "Never take two chronometers to sea. Always take one or three." - Fred Brooks, The Mythical Man-Month

Interface Testing The unit is a part of the interface (object, library) Kinds of interfaces parameter shared memory message passing Kinds of interface errors Interface misuse (e.g., sending wrong paramter) Interface misunderstanding (using interface incorrectly) timing errors (e.g.: producer, consumer speeds)

Testing Techniques Static testing code is not executed compiler helps: declarations, types, method calls, etc. Dynamic testing code is executed give input, test results

Manual Test Techniques Static techniques Reading peer review/programming: quality, readability, usability egoless programming Walkthroughs and Inspections formal team-based assessmenet Correctness proofs formal methods (e.g., preconditions, postconditions)

Coverage-based Techniques Dynamic: cover or trace the program Control-flow coverage cover all or some statements in the program Data-flow coverage trace lifetime (scope), values of variables