Testing in the Software Lifecycle Software Testing Testing in the Software Lifecycle

Component Test Explanation of term Test objects Verifies whether each software component performs correctly according to its specification Checks internal aspects and behavior of a component Also called module test, unit test, or class test depending on programming language used Test objects The software components are tested individually and isolated from all other software components of the system Isolation is to prevent external influences If a problem exists, it is definitely originating from the component

Component Test Test environment Test objects coming directly from the developer’s desk Close cooperation with development Case study Testing a class method C++ function to calculate the total price In order to test the class method a test driver is necessary: a program that calls the component under test and then receives the test object’s reaction The test driver Information could be recorded in test drivers: Test data and results Date and time of test A function that reads test cases from a file or database

Component Test Test objectives Important task is to guarantee the particular test object executes its entire functionality correctly and completely as required by the specification (functional testing) This means the behavior of input/output of test object The component is tested with a series of test cases, where each test case covers a particular input/output combination (partial functionality) What do the input/output combinations in test cases for the case study tell us? Test case 2 checks the partial functionality of “discount in the case of five or more special equipment items”. If test case 2 is executed the test object calculates an incorrect total price. Test case 2 produces a failure

Component Testing Typical defect found: Robustness testing Wrong calculation Missing or wrongly chosen program paths Eg special cases that were forgotten or misinterpreted Robustness testing Testing to determine the robustness of the software product The degree to which a component or system can function correctly in the presence of invalid inputs or stressful environment conditions Function calls and test data are used that are either known to be wrong or at least are special case not mentioned in the specification The component’s reaction should be an appropriate exception handling If there is no such exception handling, wrong inputs can trigger domain faults such as like division by zero or access through null pointer Such faults could lead to a program crash

Component Testing Such test cases are also called negative tests Example for case study (negative test) // testcase 20 price = calculate_price(-1000.00,0.00,0.00,0,0); test_ok = test_ok && (ERR_CODE == INVALID_PRICE); //testcase 30 price = calculate_price(“abc”,0.00,0.00,0,0); test_ok = test_ok && (ERR_CODE == INVALID_ARGUMENT);

Component Testing Testing nonfunctional characteristics that cannot be tested at higher test level such as efficiency and maintainability Efficiency states how efficiently a component uses computer resources Use of memory measured in kilobytes Computing time measured in milliseconds Disk or network access time measured in milliseconds Time required to execute the component’s function and algorithms Efficiency must be verified only in efficiency-critical paths or if efficiency requirements are stated in spec. Eg: testing embedded software, realtime system

Component Testing Maintainability means all the characteristics of a program that have an influence on how easy or difficult it is to change the program or to continue developing it Maintainability testing requires: Code structure Modularity Quality of comments Adherence to standards Understandability Currency of the documentation Can be done by analysis of the program text and spec Static test What are your analysis on the method in the case study?

Component Testing Test strategy Use white box testing Use debugger Tester can design test cases using their knowledge about the component’s program structures, its functions, and variable. Use debugger Observe program variables during test execution Can help checking for correct and incorrect behavior

Integration Test Explanation of terms second test level after component test Supposes that the test objects subjected to components have been tested Groups of these components are composed to form larger structural units and subsystems (integration) After assembling the components, it must be confirmed through testing that all components collaborate correctly The goal of integration testing To expose faults in the interfaces and in the interaction between integrated components Is necessary as a further test level to find collaboration and interoperability problems and isolate the causes

Integration Test Explanation of terms Test objects Component integration test is also called integration test in the small System integration test is also called integration test in the large Testing interfaces to external systems Test objects Assembled components External system or component off-the-shelf (COTS)

Integration Test Test Environment Also needs test drivers Can reuse test drivers that were used earlier for component testing Additional tools, called monitors, are required that read and log data traffic between components

Integration Test Test objectives To reveal interface and cooperation problems, as well as conflicts between integrated parts Example of integration not working Interface format may not compatible Some files are missing Developers have split the system into completely different components than were in original design Faults in data exchange A component transmits syntactically wrong or no data The communication works but the involved components interpret the received data in a different way (contradict or misinterpret spec) Data is transmitted correctly but is transmitted at the wrong time

Integration Test Is it possible to do without component testing and execute all test cases after integration is finished? It is possible but only at the risk of great disadvantages Most of failures that will occur in a test design are caused by functional faults within the individual components Because there is no suitable access to the individual component some failures cannot be provoked and many faults, cannot be found If a failure occurs in the test, it can be difficult or impossible tp locate its origin and to isolate its cause

Integration Test Integration strategies In what order should the components be integrated in order to execute the necessary testing as quickly and easily as possible? Components may be completed at different times Ad hoc strategy is to integrate the components in order in which they are ready As a component has passed the component test, check if it fits with another already tested component, or if it fits into partially integrated subsystem If so, both parts are integrated and tested Need to write stubs to help in integration Stub is a skeletal or special-purpose implementation of a software component, used to develop or test a component that calls or is otherwise dependent on it. It replaces a called component

Integration Test : Basic strategies Top-down integration The test starts with the top level component that calls other components but not called itself. Stubs replace all subordinate components Successively, integration proceeds with lower level components The higher level that has already been tested serves as test drivers Advantage Test drivers are not needed or only simple ones are required, because the higher-level components that have been tested serve as main part of the test environment Disadvantage Lower level components not yet integrated must be replaced by stubs. This can be costly

Integration Test : Basic strategies Bottom-up integration The test starts with the elementary system components that do not call further components, except for functions of operating system. Larger subsystems are assembled from the tested components and then these integrated parts are tested Advantage No stubs are needed Disadvantage Higher-level components must be simulated by test drivers

Integration Test : Basic strategies Ad hoc integration The components are being integrated in the (casual) order in which they are finished Advantage Save time, because every component is integrated as early as possible into its environment Disadvantage Stubs as well as test drivers are required

Integration Test : Basic strategies backbone integration strategy A skeletal or backbone is built into which components are gradually integrated Advantage Components can be integrated in any order Disadvantage Labour intensive skeletal or backbone is required

Integration Test : Basic strategies Top-down and bottom-up integration in their pure form can only be applied to program structured in a strictly hierarchical way A more or less individual mix of the integration strategies may be chosen Any non-incremental integration, also called big bang should be avoided Disadvantages The time leading up to the big bang is lost time that could have been spent on testing All failures will occur at the same time. It will be difficult, if not impossible, to get the system to run at all. And it will be very difficult and time-consuming to localize and correct defects

System Test Explanation of terms The next level of testing after integration test is complete Checks if the integrated product meets the specified requirements Why still need system test? In the lower test levels, the testing was done against technical specifications which is from the technical perspective of software producer. The system test looks at the system from the perspective of the customer and the future user. The tester validate whether the requirements are completely and appropriately met Many functions and system characteristics result from the interaction of all system components, consequently, they are visible on the level of the entire system and can only be observed and tested there

System Test Test object and environment Completely assembled software system Tested in an environment as similar as possible to the intended operational environment Hardware and software products should be installed on the test platform Example: a system test environment

System Test Test object and environment Example: a system test environment mainframe DB server file server application server web server workstation

System Test The system test requires a separate test environment not in customer’s operational environment Failures are likely to occur during system test and will damage customer’s operational environment such as system crashes and data loss in the production system The testers have limited or no control over parameter settings and configuration of operational environment. The test conditions can gradually change because of simultaneous running of other system with the test.

System Test Test objectives The goal of system test is to validate whether the complete system meets the specified functional and nonfunctional requirements and how well it does that Failures from incorrect, incomplete, or inconsistent implementation of requirements should be detected Requirements that are undocumented or have been forgotten should be identified

System Test Problems in system test Unclear system requirements Use exploratory testing Missed decision Different people have completely different views and idea Project may fail If no requirements, then not likely to meet the implicit requirement of the customer

Acceptance test Focuses on customer’s perspective and judgment Involves customer to test Can be executed within lower test levels or distributed over several test levels COTS software product may be acceptance tested when it is installed or integrated Acceptance test of a component’s usability may be done during component test Acceptance test of new functionality may come before system testing using a prototype

Acceptance test How much acceptance test? Varies considerably depending upon application risk The risk is high and a full acceptance test required for customer specific software The risk is low and not full acceptance test if an acquisition of a standard product Acceptance test consists of installing the system and running a few representative use cases. If the system is expected to cooperate with other systems in a new way, at least the interoperation should be tested

Acceptance test: Typical forms Testing for acceptance according to contract For customer specific software customer in cooperation with vendor will perform the test according to the contract Customer, based on the results of testing, considers whether the ordered software system is free of deficiencies and whether the development contract or the service defined by the contract has been accomplished The test criteria are acceptance criteria determined in the development contract These criteria must be formulated clearly and explicitly Also regulation (government, legal, or safety) must be adhered to Acceptance test cases be designed by, or at least reviewed by customer

Acceptance test: Typical forms Testing for acceptance according to contract Software producer checks these criteria within his own system test and rerun the test cases relevant for acceptance System test takes place in producer’s environment, acceptance test takes place in the customer’s actual operational environment Be careful, a test case that works correctly during system test may suddenly fail.

Acceptance test: Typical forms Testing for user acceptance Recommended if the customer and the user are different individuals Different user groups usually have different expectations Need to get acceptance from all user groups Customer organizes tests, selects test cases based on business processes and typical use scenarios Allow a number of representatives from future users to examine prototypes at an early stage of the project to avoid acceptance problems

Acceptance test: Typical forms Operational (acceptance) testing Assures the acceptance of the system by the system administrators May include the testing of backup/restore cycle, disaster recovery, user management, maintenance tasks, and checks of security vulnerabilities

Acceptance test: Typical forms Field test (alpha and beta testing) Choose to carry out field test if software is supposed to run in many different operational environments To identify the influences from users’ environment that are not entirely known or that are not specified, and to eliminate them if necessary Producer delivers stable prerelease versions of the software to pre-selected customers Adequately represent the market Whose operational environment appropriately cover possible environments

Acceptance test: Typical forms Field test (alpha and beta testing) The customer then either Run test scenarios prescribed by the producer Run the product on a trail basis under realistic conditions Customer gives feedback to the producer about the problems encountered General comments and impression Alpha testing Test carried out at the producer’s location Beta testing Test carried out at the customer’s site

Testing New Product Versions Software maintenance Typical problems found: The system is run under new operating conditions that were not predictable and were not planned The customers express new wishes Functions are necessary for rarely occurring special cases that were not anticipated Crashes that happen rarely or only after very long uptime are reported. These crashes are often caused by external influences After its deployment, every software requires certain correction and improvement

Testing New Product Versions Software maintenance The strategy is: Anything new or changed should be tested The remainder of the system should be regression tested If only its environment is changed, maintenance testing is necessary Migration from one platform to another testing should repeat the operational tests within the new environment

Testing New Product Versions Release development Product is not finished with the release of the first version Changes and extensions that are intended from beginning A planned change of a neighbor system Functionality that had been planned from the beginning, but could not be implemented as early as intended Extensions become necessary in the course of a planned market expansion The strategy is: Anything new or changed should be tested The remainder of the system should be regression tested

Generic type of testing : Functional Testing Includes all kind of tests which verify a system’s input- output behavior Black box testing methods are used to design functional test cases Test bases are the functional requirements In requirement-based testing The released requirements are used as the basis for testing For each requirement, at least one test case is designed and documented in the test specification (usually more than one) The test specification is then reviewed Used in system testing and acceptance testing

Generic type of testing : Nonfunctional Testing Testing attributes of the functional behavior, or the attributes of the system as a whole “how well” or with what quality the system should carry out its function Nonfunctional characteristics that should be considered in tests (usually in system testing) Load test Measuring of system behavior for increasing system loads (e.g., the number of users that work simultaneously, number of transactions) Performance test Measuring of the processing speed and response time for a particular use cases, usually dependent on increasing load

Generic type of testing : Nonfunctional Testing Nonfunctional characteristics that should be considered in tests (usually in system testing) Volume test Observation of the system behavior dependent on the amount of data (e.g., processing of very large files) Stress test Observation of the system behavior when it is overloaded Testing of security against unauthorized access, denial of services etc Stability or reliability test during permanent operation (e.g. mean time between failure or failure rate with a given user profile)

Generic type of testing : Nonfunctional Testing Nonfunctional characteristics that should be considered in tests (usually in system testing) Robustness test Measuring the system’s response to operating error, or wrong programming, or hardware failure etc, as well as examination of exception handling and recovery Testing of compatibility and data conversion Examination of compatibility to given system, import/export of data Testing of different configuration of the system, e.g., different version of OS, user interface language, hardware platform etc. (back-to-back testing) Usability test Examination of the ease of learning the system, ease and efficiency of operation, understandability of the system output

Generic type of testing : Nonfunctional Testing Nonfunctional characteristics that should be considered in tests (usually in system testing) Checking of the documentation for compliance with system behavior (e.g., user manual and GUI Checking of maintainability Assessing the understandability of the system documentation and whether it is up to date; checking if the system has a modular structure etc.

Testing related to Changes and Regression Testing