System Test Methods TESTTEME The Test Challenge Bottom Up Testing Strategy Integration Test System Test Types of Testing Unit Test = Code-based Testing Integration Test = Data-based Testing System Test = Requirements-based Testing Testing Interdependencies Software Testing Strategies Hierarchical Software Systems BIG-BANG Strategy TOP-DOWN Strategy BOTTOM-UP Strategy INSIDE-OUT Strategy OUTSIDE-IN Strategy BRANCHWISE Strategy Comparison of the Test Strategies Program Test Methods Control Flow Testing (White-Box) Data Flow Testing (Grey-Box) Functional Testing (Black-Box) Testing Heuristics Roll of Testing in the Life Cycle

Budget-, Time- and Personnel Constraints Great Diversity of SW- Environments Applications are becoming increasingly large and complex SW-Developers are neither trained nor motivated to test Testers are willing but incapable Lack of a testing culture The Testing Challenge TESTTEME-1

„How do you eat an elephant... bit by bit“ African Proverb Class Test –Test of each individual Class as a standalone Unit Integration Test –Test of several related classes and components System Test –Test of subsystems and the entire system Acceptance Test –Test by the User in a live environment Bottom Up Testing Strategy TESTTEME-2

„Integration testing exercises interfaces among units within the specified scope to demonstrate that the units are collectively interoperable“ Binder, 1999 Classes Class Hierarchies Packages of related Classes = Components Subsystems Client/Server = Frontend/Backend Entire Application Integration Test TESTTEME-3

The Whole is more than the Sum of the Parts –„Individual verification of components cannot garantee a correctly functioning system“ - Binder, 1999 Demonstrating the required Functionality –„If you do not have written objectives for your product, i.e a specification, or if your objectives are unmeasurable, then you cannot perform system testing“ - Myer, 1976 Determining when to release a product ?! –Probability of remaining Errors.. System Test TESTTEME-4

Classes under Test Class Flattener Test Driver Test Stubs Test Methods Integration Testing Specified Arguments Message Generator Input Messages Components under Test Return Messages Message Validator Unit Testing Expected Return Values ReplayCapture Web Pages Web Client Web Server Data System Testing Data Generator Data Validator RetrievalUpdates Types of Testing TESTTEME-5

A Test is always a test against Something TESTTEME-6 Concept System Test Cases Test Results Test Cycle new System Test Cases Test Results Test Cycle current System Verification = Comparison In order to verify a system, the requirements must be independent of the code. The task is to test the system against the requirements by extracting test cases from the requirements specification. Verification implies demonstrating that the system fulfills it‘s requirements. Once a system exists, every new version can be tested against the previous one. In this case the test cases can be extracted from the existing system and enhanced by test cases taken from the change requests. The new version is verified against the old one plus the new requirements. Initial TestingRegession Testing

Code based Testing TESTTEME-7 Code Test inputsTest outcomes SpecificationExpected outcomes Interfaces/Databases Predicate Paths Specified Paths Actual Paths compare Paths Corresponds to a Unit Test (White-Box)

Unit Test = Code-based Testing TESTTEME-8 Block IF THEN ELSE DO WHILECASE Block IF THEN = Testpoint WHITE-BOX-Method (Modultest) Test Object:

Data based Testing TESTTEME-9 CodeTest inputsTest outcomes SpecificationExpected outcomes compare Data Representative Values Interfaces/Files/Tables Specified Results Interfaces/Databases Corresponds to an Integration Test (Grey-Box)

Client Server Data- server Interface Interface SimulationInterface Validation GREY-BOX Method (Program Test) Input Data Output Data Integration Test = Data-based Testing TESTTEME-10

Specification based Testing TESTTEME-11 CodeTest inputsTest outcomes Specification Expected outcomes compare Files/Tables/Panels Specified Results Expected Values Interfaces/Databases Corresponds to a System Test (Black-Box) Outputs

SystemTest = Requirements-based Testing TESTTEME-12 Test Object Web Pages Masks Files Databases Parameters (Arguments) Messages Web Pages Masks Files Databases Reports (Results) Messages SYSTEM INPUTS SYSTEM OUTPUTS Application Function BLACK-BOX Method (System Test)

Integration Testing Strategies TESTTEME-13 BIG-BANG TOP-DOWN BOTTOM-UP INSIDE-OUT OUTSIDE-IN BRANCH-WISE

Layered Software Systems TESTTEME-14 Framework GUI- Components Error handling Module Print Module XML Output Module XML Input Module SQL Acess Module Order Art-3 Module Order Art-4 Module Order Art-1 Module Order Art-2 Module Order Entry Processing System Sample of a three level Program Service Level Logic Level Presentation Level

BIG-BANG Strategy TESTTEME-15 GUI Components Error handling Module Order Art-3 Module Order Art-4 Module Order Art-1 Module Order Art-2 Module Print Module XML Output Module XML Input Module SQL Access Module

TOP-DOWN Strategy TESTTEME-16 Framework GUI Components Stub

BOTTOM-UP Strategy TESTTEME-17 Framework GUI Components Error handling Module Print Module XML Output Module XML Input Module SQL Access Module Driver Test-driven Development

INSIDE-OUT Strategy TESTTEME-18 Driver Stub Order Art-3 Module Order Art-4 Module Order Art-1 Module Order Art-2 Module

OUTSIDE-IN Strategy TESTTEME-19 Framework GUI Components Error Handling Module Print Module XML Output Module XML Input Module SQL Access Module Driver Stub

BRANCHWISE Strategy TESTTEME-20 Framework GUI- Components Stub Print Module Stub XML Input Module SQL Access Module Stub Order Art-1 Module Stub (Thread Testing) One Transaction at a time Is tested through to end Straight thru testing

Comparison of Test Strategies TESTTEME-21 Point of View TOP-DOWN BOTTOM-UP INSIDE-OUT OUTSIDE-IN BRANCH-WISE Division of Labor poor good poor good poor Controllability poor good poor medium medium Test Aides required Stubs Drivers Stub&Drivers Stub&Drivers Stubs Test Case Definition difficult easy medium easy medium Test Visibility poor good medium good poor Duration of Test medium long long short short According to G. Myers: Software Reliability Principles and Practices

Principle SoftwareTest Approaches TESTTEME-22 Control Flow Testing  Statement Coverage  Branch Coverage  Path Coverage Data Flow Testing  Repreäsentative value coverage  Boundary value coverage  Equivalence Class Coverage  Data Relation Coverage  Cause & Effect Analysis  State Coverage Functional Testing  Result coverage = all desired results  Rule coverage = all relevant rules  Input oriented = all possible inputs

Control-Flow Testing (White-Box) TESTTEME-23 Entry Exit (1) (2)(3)(5)(6) (7)(8) (9) (10) (4) (11) (12) ( ) = Control- branch Paths = 1, 2, 3, 4, 6, 11 1, 2, 3, 5, 6, 11 1, 2, 7, 8, 10, 11 1, 2, 7, 9, 10, 11 1, 12,

Data-Flow Testing (Grey-box) TESTTEME-24 Program Inputs/ArgumentsOutputs/Results Predicates Representative Values Boundary Values Equivalence Classes Discrete Values Value Domains Functional Results Relations True/False Cause/Effect

Functional Testing (Black-Box) TESTTEME-25 Software SystemSystem Specification Arguments Rules Results Inputs Paths Outputs

Testing Heuristics TESTTEME-26 The specification of expected results is an essential part of the test. Programmers are not in a position to test their own programs. The results of each test case must be verified. All exception conditions should be tested and confirmed. It is not enough to confirm that a program does what it should do, but also to confirm that it does not do what it should not do. Test cases must be repeatable. Never assume there are no errors, software is always erroneoius. Errors tend to cluster. If an error is found, other errors are likely to be found around it. A good test case is a case with a high probability of discovering an error. The goal of testing is to find errors. A test with no defined end criteria is no test