1. Testing in the Software Lifecycle
Software Testing
Testing in the Software Lifecycle

2. Component Test Explanation of term Test objects
Verifies whether each software component performs correctly according to its specification
Checks internal aspects and behavior of a component
Also called module test, unit test, or class test depending on programming language used
Test objects
The software components are tested individually and isolated from all other software components of the system
Isolation is to prevent external influences
If a problem exists, it is definitely originating from the component

3. Component Test Test environment
Test objects coming directly from the developer’s desk
Close cooperation with development
Case study
Testing a class method
C++ function to calculate the total price
In order to test the class method a test driver is necessary: a program that calls the component under test and then receives the test object’s reaction
The test driver
Information could be recorded in test drivers:
Test data and results
Date and time of test
A function that reads test cases from a file or database

4. Component Test Test objectives
Important task is to guarantee the particular test object executes its entire functionality correctly and completely as required by the specification (functional testing)
This means the behavior of input/output of test object
The component is tested with a series of test cases, where each test case covers a particular input/output combination (partial functionality)
What do the input/output combinations in test cases for the case study tell us?
Test case 2 checks the partial functionality of “discount in the case of five or more special equipment items”. If test case 2 is executed the test object calculates an incorrect total price. Test case 2 produces a failure

5. Component Testing Typical defect found: Robustness testing
Wrong calculation
Missing or wrongly chosen program paths
Eg special cases that were forgotten or misinterpreted
Robustness testing
Testing to determine the robustness of the software product
The degree to which a component or system can function correctly in the presence of invalid inputs or stressful environment conditions
Function calls and test data are used that are either known to be wrong or at least are special case not mentioned in the specification
The component’s reaction should be an appropriate exception handling
If there is no such exception handling, wrong inputs can trigger domain faults such as like division by zero or access through null pointer
Such faults could lead to a program crash

6. Component Testing Such test cases are also called negative tests
Example for case study (negative test)
// testcase 20
price = calculate_price( ,0.00,0.00,0,0);
test_ok = test_ok && (ERR_CODE == INVALID_PRICE);
//testcase 30
price = calculate_price(“abc”,0.00,0.00,0,0);
test_ok = test_ok && (ERR_CODE == INVALID_ARGUMENT);

7. Component Testing
Testing nonfunctional characteristics that cannot be tested at higher test level such as efficiency and maintainability
Efficiency states how efficiently a component uses computer resources
Use of memory measured in kilobytes
Computing time measured in milliseconds
Disk or network access time measured in milliseconds
Time required to execute the component’s function and algorithms
Efficiency must be verified only in efficiency-critical paths or if efficiency requirements are stated in spec.
Eg: testing embedded software, realtime system

8. Component Testing
Maintainability means all the characteristics of a program that have an influence on how easy or difficult it is to change the program or to continue developing it
Maintainability testing requires:
Code structure
Modularity
Quality of comments
Adherence to standards
Understandability
Currency of the documentation
Can be done by analysis of the program text and spec
Static test
What are your analysis on the method in the case study?

9. Component Testing Test strategy Use white box testing Use debugger
Tester can design test cases using their knowledge about the component’s program structures, its functions, and variable.
Use debugger
Observe program variables during test execution
Can help checking for correct and incorrect behavior

10. Integration Test Explanation of terms
second test level after component test
Supposes that the test objects subjected to components have been tested
Groups of these components are composed to form larger structural units and subsystems (integration)
After assembling the components, it must be confirmed through testing that all components collaborate correctly
The goal of integration testing
To expose faults in the interfaces and in the interaction between integrated components
Is necessary as a further test level to find collaboration and interoperability problems and isolate the causes

11. Integration Test Explanation of terms Test objects
Component integration test is also called integration test in the small
System integration test is also called integration test in the large
Testing interfaces to external systems
Test objects
Assembled components
External system or component off-the-shelf (COTS)

12. Integration Test Test Environment Also needs test drivers
Can reuse test drivers that were used earlier for component testing
Additional tools, called monitors, are required that read and log data traffic between components

13. Integration Test Test objectives
To reveal interface and cooperation problems, as well as conflicts between integrated parts
Example of integration not working
Interface format may not compatible
Some files are missing
Developers have split the system into completely different components than were in original design
Faults in data exchange
A component transmits syntactically wrong or no data
The communication works but the involved components interpret the received data in a different way (contradict or misinterpret spec)
Data is transmitted correctly but is transmitted at the wrong time

14. Integration Test
Is it possible to do without component testing and execute all test cases after integration is finished?
It is possible but only at the risk of great disadvantages
Most of failures that will occur in a test design are caused by functional faults within the individual components
Because there is no suitable access to the individual component some failures cannot be provoked and many faults, cannot be found
If a failure occurs in the test, it can be difficult or impossible tp locate its origin and to isolate its cause

15. Integration Test Integration strategies
In what order should the components be integrated in order to execute the necessary testing as quickly and easily as possible?
Components may be completed at different times
Ad hoc strategy is to integrate the components in order in which they are ready
As a component has passed the component test, check if it fits with another already tested component, or if it fits into partially integrated subsystem
If so, both parts are integrated and tested
Need to write stubs to help in integration
Stub is a skeletal or special-purpose implementation of a software component, used to develop or test a component that calls or is otherwise dependent on it. It replaces a called component

16. Integration Test : Basic strategies
Top-down integration
The test starts with the top level component that calls other components but not called itself.
Stubs replace all subordinate components
Successively, integration proceeds with lower level components
The higher level that has already been tested serves as test drivers
Advantage
Test drivers are not needed or only simple ones are required, because the higher-level components that have been tested serve as main part of the test environment
Disadvantage
Lower level components not yet integrated must be replaced by stubs. This can be costly

17. Integration Test : Basic strategies
Bottom-up integration
The test starts with the elementary system components that do not call further components, except for functions of operating system. Larger subsystems are assembled from the tested components and then these integrated parts are tested
Advantage
No stubs are needed
Disadvantage
Higher-level components must be simulated by test drivers

18. Integration Test : Basic strategies
Ad hoc integration
The components are being integrated in the (casual) order in which they are finished
Advantage
Save time, because every component is integrated as early as possible into its environment
Disadvantage
Stubs as well as test drivers are required

19. Integration Test : Basic strategies
backbone integration strategy
A skeletal or backbone is built into which components are gradually integrated
Advantage
Components can be integrated in any order
Disadvantage
Labour intensive skeletal or backbone is required

20. Integration Test : Basic strategies
Top-down and bottom-up integration in their pure form can only be applied to program structured in a strictly hierarchical way
A more or less individual mix of the integration strategies may be chosen
Any non-incremental integration, also called big bang should be avoided
Disadvantages
The time leading up to the big bang is lost time that could have been spent on testing
All failures will occur at the same time. It will be difficult, if not impossible, to get the system to run at all. And it will be very difficult and time-consuming to localize and correct defects

21. System Test Explanation of terms
The next level of testing after integration test is complete
Checks if the integrated product meets the specified requirements
Why still need system test?
In the lower test levels, the testing was done against technical specifications which is from the technical perspective of software producer. The system test looks at the system from the perspective of the customer and the future user. The tester validate whether the requirements are completely and appropriately met
Many functions and system characteristics result from the interaction of all system components, consequently, they are visible on the level of the entire system and can only be observed and tested there

22. System Test Test object and environment
Completely assembled software system
Tested in an environment as similar as possible to the intended operational environment
Hardware and software products should be installed on the test platform
Example: a system test environment

23. System Test Test object and environment
Example: a system test environment
mainframe
DB server file server application server
web server
workstation

24. System Test
The system test requires a separate test environment not in customer’s operational environment
Failures are likely to occur during system test and will damage customer’s operational environment such as system crashes and data loss in the production system
The testers have limited or no control over parameter settings and configuration of operational environment. The test conditions can gradually change because of simultaneous running of other system with the test.

25. System Test Test objectives
The goal of system test is to validate whether the complete system meets the specified functional and nonfunctional requirements and how well it does that
Failures from incorrect, incomplete, or inconsistent implementation of requirements should be detected
Requirements that are undocumented or have been forgotten should be identified

26. System Test Problems in system test Unclear system requirements
Use exploratory testing
Missed decision
Different people have completely different views and idea
Project may fail
If no requirements, then not likely to meet the implicit requirement of the customer

27. Acceptance test Focuses on customer’s perspective and judgment
Involves customer to test
Can be executed within lower test levels or distributed over several test levels
COTS software product may be acceptance tested when it is installed or integrated
Acceptance test of a component’s usability may be done during component test
Acceptance test of new functionality may come before system testing using a prototype

28. Acceptance test How much acceptance test?
Varies considerably depending upon application risk
The risk is high and a full acceptance test required for customer specific software
The risk is low and not full acceptance test if an acquisition of a standard product
Acceptance test consists of installing the system and running a few representative use cases.
If the system is expected to cooperate with other systems in a new way, at least the interoperation should be tested

29. Acceptance test: Typical forms
Testing for acceptance according to contract
For customer specific software customer in cooperation with vendor will perform the test according to the contract
Customer, based on the results of testing, considers whether the ordered software system is free of deficiencies and whether the development contract or the service defined by the contract has been accomplished
The test criteria are acceptance criteria determined in the development contract
These criteria must be formulated clearly and explicitly
Also regulation (government, legal, or safety) must be adhered to
Acceptance test cases be designed by, or at least reviewed by customer

30. Acceptance test: Typical forms
Testing for acceptance according to contract
Software producer checks these criteria within his own system test and rerun the test cases relevant for acceptance
System test takes place in producer’s environment, acceptance test takes place in the customer’s actual operational environment
Be careful, a test case that works correctly during system test may suddenly fail.

31. Acceptance test: Typical forms
Testing for user acceptance
Recommended if the customer and the user are different individuals
Different user groups usually have different expectations
Need to get acceptance from all user groups
Customer organizes tests, selects test cases based on business processes and typical use scenarios
Allow a number of representatives from future users to examine prototypes at an early stage of the project to avoid acceptance problems

32. Acceptance test: Typical forms
Operational (acceptance) testing
Assures the acceptance of the system by the system administrators
May include the testing of backup/restore cycle, disaster recovery, user management, maintenance tasks, and checks of security vulnerabilities

33. Acceptance test: Typical forms
Field test (alpha and beta testing)
Choose to carry out field test if software is supposed to run in many different operational environments
To identify the influences from users’ environment that are not entirely known or that are not specified, and to eliminate them if necessary
Producer delivers stable prerelease versions of the software to pre-selected customers
Adequately represent the market
Whose operational environment appropriately cover possible environments

34. Acceptance test: Typical forms
Field test (alpha and beta testing)
The customer then either
Run test scenarios prescribed by the producer
Run the product on a trail basis under realistic conditions
Customer gives feedback to the producer
about the problems encountered
General comments and impression
Alpha testing
Test carried out at the producer’s location
Beta testing
Test carried out at the customer’s site

35. Testing New Product Versions
Software maintenance
Typical problems found:
The system is run under new operating conditions that were not predictable and were not planned
The customers express new wishes
Functions are necessary for rarely occurring special cases that were not anticipated
Crashes that happen rarely or only after very long uptime are reported. These crashes are often caused by external influences
After its deployment, every software requires certain correction and improvement

36. Testing New Product Versions
Software maintenance
The strategy is:
Anything new or changed should be tested
The remainder of the system should be regression tested
If only its environment is changed, maintenance testing is necessary
Migration from one platform to another
testing should repeat the operational tests within the new environment

37. Testing New Product Versions
Release development
Product is not finished with the release of the first version
Changes and extensions that are intended from beginning
A planned change of a neighbor system
Functionality that had been planned from the beginning, but could not be implemented as early as intended
Extensions become necessary in the course of a planned market expansion
The strategy is:
Anything new or changed should be tested
The remainder of the system should be regression tested

38. Generic type of testing : Functional Testing
Includes all kind of tests which verify a system’s input- output behavior
Black box testing methods are used to design functional test cases
Test bases are the functional requirements
In requirement-based testing
The released requirements are used as the basis for testing
For each requirement, at least one test case is designed and documented in the test specification (usually more than one)
The test specification is then reviewed
Used in system testing and acceptance testing

39. Generic type of testing : Nonfunctional Testing
Testing attributes of the functional behavior, or the attributes of the system as a whole
“how well” or with what quality the system should carry out its function
Nonfunctional characteristics that should be considered in tests (usually in system testing)
Load test
Measuring of system behavior for increasing system loads (e.g., the number of users that work simultaneously, number of transactions)
Performance test
Measuring of the processing speed and response time for a particular use cases, usually dependent on increasing load

40. Generic type of testing : Nonfunctional Testing
Nonfunctional characteristics that should be considered in tests (usually in system testing)
Volume test
Observation of the system behavior dependent on the amount of data (e.g., processing of very large files)
Stress test
Observation of the system behavior when it is overloaded
Testing of security against unauthorized access, denial of services etc
Stability or reliability test during permanent operation (e.g. mean time between failure or failure rate with a given user profile)

41. Generic type of testing : Nonfunctional Testing
Nonfunctional characteristics that should be considered in tests (usually in system testing)
Robustness test
Measuring the system’s response to operating error, or wrong programming, or hardware failure etc, as well as examination of exception handling and recovery
Testing of compatibility and data conversion
Examination of compatibility to given system, import/export of data
Testing of different configuration of the system, e.g., different version of OS, user interface language, hardware platform etc. (back-to-back testing)
Usability test
Examination of the ease of learning the system, ease and efficiency of operation, understandability of the system output

42. Generic type of testing : Nonfunctional Testing
Nonfunctional characteristics that should be considered in tests (usually in system testing)
Checking of the documentation for compliance with system behavior (e.g., user manual and GUI
Checking of maintainability
Assessing the understandability of the system documentation and whether it is up to date; checking if the system has a modular structure etc.

43. Testing related to Changes and Regression Testing