1. Fundamental Test Process

2. Introduction Software Engineering (or Development) employs
Software Lifecycle Models (e.g. Waterfall, V, etc.)
Development processes (e.g. OOA, OOD, OOP)
These models define a systematic way to achieve an orderly way of working during a project.
A phase completion (aka a milestone) is achieved when the required documents are completed and conform to the given quality criteria.
Testing appears in each Software Lifecycle Model, but with very different meanings and different extent.

3. Introduction
The description of phases (and activities/tasks within the phases) does not provide detailed instruction on how to perform tests in software projects.
Therefore a more detailed process for the testing tasks is required.
This process is known as the Fundamental Test Process.
The Fundamental Test Process identifies the testing activities/tasks that should be undertaken (in the context of any software lifecycle model.)

4. Fundamental Test Process
Although logically sequential, the activities or tasks in the test process may overlap or take place concurrently.
The activities are:
Test Planning and Control
Test Analysis and Design
Test Implementation and Execution
Evaluation of Test Exit Criteria and Reporting
Post Testing (Closure) Activities

5. Fundamental Test Process
Begin
Planning and
Control
Analysis and Design
Implementation and Execution
Evaluation of the Test Exit Criteria
Post Testing Activities
End

6. Test Planning and Control
A task as large as testing should not take place without a plan.
Planning of the test process should begin at the beginning of the software development project.
As well as planning, the progress of the project plan(s) must be regularly checked (or monitored) and updated (controlled).

7. Test Planning and Control
The objectives of testing must be defined and agreed.
Resources should be estimated:
Which people are required,
For the execution of which tasks, and
When?
These questions and more are answered as part of the planning process.

8. Test Planning and Control
Test control is
the monitoring of the test activities,
comparing what actually happens during the project with the plan,
reporting status of deviations from the plan, and
taking any actions to meet the objectives in the new situation.
The Test Plan must be continuously be updated, taking into account the feedback from monitoring and control.

9. Test Planning and Control
The main task of planning is to determine the Test Strategy.
Since exhaustive testing is not possible, a test strategy must be put in place to determine what is to be tested and what is not to be tested.
The goal of a test strategy is to ensure the testing of the “right” parts of the software system under test.

10. Test Planning and Control
Example of a Test Strategy:
Risk Assessment or Risk Based Testing.
The intensity or amount of testing undertaken depends very much on which test techniques are employed and the test coverage that must be achieved.
Test coverage serves as a test exit criterion, which helps in determining if your testing has been completed as planned with respect to the test strategy.

11. Test Planning and Control
Examples of exit criterion include:
Fulfilment of all customer requirements,
All functions/methods be at least once, or
80% of all transactions in a system are executed.
Once all test exit criteria are defined, their values are examined after executing the planned test cases to decide if the test process can be finished.

12. Test Planning and Control
As software projects run under severe time pressure, test cases should be prioritised.
Prioritisation of tests guarantees that the most critical or important parts of the system are tested first, in case time constraints do not allow the execution of all planned tests.

13. Test Planning and Control
Selection and acquisition of test tools must be initiated early.
If frameworks are to be employed, such as JUnit, their usage in the project must be planned and engineers informed of the need to use the framework.

14. Test Analysis and Design
First step is to review the specification of what should be tested.
The specification may not be clear or concise to develop test cases.
If the testability of the specification is insufficient, rework of the specification has to be done.

15. Test Analysis and Design
Determining the preconditions and requirements to test case design is based on:
An analysis of the specification,
The expected behaviour, and
The nature/structure of the Test Object.
The test strategy (identified in the test plan) defines which test techniques shall be employed.

16. Test Analysis and Design
Appropriate test cases are then developed
using the test techniques specified in the test plan, as well as
Techniques chosen based on an analysis of the possible complexity in the test object.
Design test cases consists of two steps:
Definition of Logical Test Cases, which are
Translated into physical/concrete test cases, where actual inputs are chosen.

17. Test Analysis and Design
The development or implementation of physical test cases is part of the next phase in the fundamental test process – test implementation.
The Test Basis (i.e. the specification of the object under test) guides the selection of logical test cases with each test technique.

18. Test Analysis and Design
Test cases can be determined from
the test object’s specification (black box test design technique), or
Created by analyzing the source code (white box test design technique).
Test case design (or specification) can take place at different points or time in the software development cycle.

19. Test Analysis and Design
A test has the following characteristics:
an object under test
a test objective
a definition of the inputs
a definition of expected outputs
a definition of the test environment.
When a test is executed (in a later phase of the test process):
an actual output is generated/observed
the tester determines if the output is correct.

20. Test Analysis and Design
Specifications define what the software is required to do.
Without specifications, developers cannot build, testers cannot test.
Testers need requirements to:
identify what needs to be tested
compare test results with requirements

21. Test Analysis and Design
If an expected result is not defined before a test is executed
an erroneous result may be interpreted as the correct result
Expected results must be defined before test execution, derived from a specification document, design document, or code.
(If there is a difference between Expected and Actual Outputs, there may be a fault in the software and this should be investigated.)

22. Test Analysis and Design – An Example
Specification: A company orders an application that should calculate the Holiday Bonus of its employees depending on the length of service to the company. The requirement description states: “Employees with a length of service greater than 3 years will get 50% of their monthly salary as a bonus. Employees with more than 5 years service will get 75% of salary as bonus. Employees with the company longer than 8 years will get a 100% bonus.”

23. Test Analysis and Design – An Example
The specification could be summarised as follows:
Length of Service <= 3, Bonus = 0%
3 < Length of Service <= 5, Bonus = 50%
5 < Length of Service <= 8, Bonus = 75%
Length of Service > 8, Bonus = 100%

24. Test Analysis and Design – An Example
Based on the specification, it is possible to create the following logical test cases:
Test Case Number
Input x
(Length of Service)
Expected Result
(% Bonus) 1
X <= 3 2
3 < X <= 5 50 3
5 < X <= 8 75 4
X > 8
100

25. Test Analysis and Design – An Example
To execute the test cases, the logical test cases must be converted in physical or concrete test cases. i.e. concrete input data (values) must be defined.
Preconditions are not specified for the test cases below.
Test Case Number
Input x
(Length of Service)
Expected Result
(% Bonus) 1 2 4 50 3 7 75 19
100

26. Test Analysis and Design – An Example
This is example just demonstrates the differences between logical and concrete test cases.
No explicit test techniques were employed to identify these test cases.
4 test cases does not adequately test this requirement. (e.g. What happens if a user inputs a negative number?)
(We will discuss creating systematic test cases with appropriate test techniques later.)

27. Test Analysis and Design
In parallel to specifying or designing your test cases it is important to set-up the test environment required execute the test object.

28. Test Implementation and Execution
In this phase,
The concrete test cases are implemented (by either writing test code or a test procedure),
The test cases are executed (on the appropriate test environment) and logged, and
Faults found logged.

29. Test Implementation and Execution
Test cases should be grouped into test suites for efficient test execution.
Test case execution should be prioritised. Giving priority means that the most important tests are executed first and thus important problems are found and corrected early.
Specific test harnesses (drivers, simulators, etc.) must be setup and checked that they are functioning correctly.
Test execution may be manual or automated.
Test execution must be exactly and completely logged, including which tests passes and failed. This also proves that the planned test strategy was actually executed.

30. Test Implementation and Execution
The information belonging to a test case or test run must be maintained in such a way that it is possible to repeat the test later with the same input data and conditions.
If a difference is found between the expected and actual results for a test, it must be determined if this is really a failure and not an issue with something else (e.g. a problem with the test environment).

31. Test Implementation and Execution
If a failure is confirmed then it (may be) logged as a Problem Report in the Incident Management system, is fixed, and re-tested to ensure that the fault has been fixed.
(Note that fixing faults may introduced new faults into the software.)
(Fixing faults may require the generation of some additional test cases.)

32. Test Implementation and Execution
In many projects, there is not enough time to execute all the specified test cases – when that happens, make a reasonable selection of test cases to ensure as many critical failures are detected.

33. Evaluation of the Test Exit Criteria and Reporting
The test object is assessed against set objectives – the test exit criteria specified earlier in the process.
This may result in normal termination of the tests if all criteria are met, or it may be decided that additional tests should be run or the criteria re-assessed and limits adjusted.

34. Evaluation of the Test Exit Criteria and Reporting
For example, it could be stated that a test is sufficient if 75% of the statements of the test object were executed during test execution.
Appropriate tools should be used to collect such measures or metrics in order to determine if a test has met its exit criteria.
(More examples of test coverage criteria will be identified later when different test techniques are discussed.)

35. Evaluation of the Test Exit Criteria and Reporting
In addition to test coverage criteria, other criteria can be used to define the end of a test:
Failure Rate
Failure Severity.
Have all the planned test cycles been executed? Are additional test cycles required?

36. Evaluation of the Test Exit Criteria and Reporting
In practice the end of a test is determined by factors that have no direct connection to the test: time and cost.
If these factors lead to stopping the test activities, it is because not enough resources were provided in the project plan or the effort for an adequate test was underestimated.
When the test criteria are fulfilled (or their nonfulfillment is determined), a test summary report should be written for the relevant stakeholders.

37. Test Closure Activities
Experience gathered during the test work should be analysed and made available for further projects.
Critical evaluation of the employed test process should be undertaken.
The developed testware should be handed over to the maintenance team for the developed product.