1. CompSci 280 S2 2107 Introduction to Software Development
If debugging is the process of removing software bugs, then programming must be the process of putting them in. (Edsger Dijkstra)
CompSci 280 S Introduction to Software Development
Software Testing

2. Today’s Agenda Topics: Reading: Introduction
Verification vs Validation
Testing
Unit Testing
Reading:
Software Engineering 10th Edition, Ian Somerville
Chapter 8: Software Testing
Lecture19

3. 1.Introduction Program testing
Testing is intended to show that a program does what it is intended to do and to discover program defects before it is put into use.
When you test software, you execute a program using artificial data.
You check the results of the test run for errors, anomalies or information about the program’s non-functional attributes.
Can reveal the presence of errors NOT their absence.
Lecture19

4. 1.Introduction Testing process goals
Validation testing
To demonstrate to the developer and the system customer that the software meets its requirements
There should be at least one test for every requirement in the requirements document.
A successful test shows that the system operates as intended.
Defect testing
To discover faults or defects in the software where its behaviour is incorrect or not in conformance with its specification
Rooting out undesirable system behavior such as system crashes, unwanted interactions with other systems, incorrect computations and data corruption.
A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.
Lecture19

5. 2.Verification vs Validation
Verification: "Are we building the product right”.
The software should conform to its specification.
that every activity has been carried out correctly
that the thing has been built right
Validation: "Are we building the right product”.
The software should do what the user really requires.
the right thing has been built
V&V is carried out in parallel with the software/system development process.
V&V activities include traceability analysis, evaluation, review, inspection, assessment, and testing.
Lecture19

6. 2. Verification & Validation Verification
Are we building the product right
Verification is achieved through:
Static analysis of documentation (refer figure: Static Verification)
Evaluation of work products in the context of reviews
i.e. Static detection (before runtime)
Code reviews can help to detect faults
Traceability analysis
Effective but costly!!!
Dynamic testing
Observing the behaviour software executing on a computer (refer figure: Dynamic Testing)
Testing, i.e. by running code
Requires a (partially) running system already
Usually in the implementation stage of a project
Lecture19

7. 2. Verification & Validation Verification
Are we building the product right
Static Verification:
Requirements trace to business needs
Code implements all design elements
Design implements all requirements
Lecture19

8. 2. Verification & Validation The V Model
It is a variation of the waterfall process that includes (some) verification and validation
Does the system achieve the business goals?
Does the system satisfy the end user’s requirements?
Does the integrated system behaves per the overall design?
Does individual software modules behave as per the design?
Dynamic Testing
Lecture19

9. 2. Verification & Validation
Are we building the right product?
Validation demonstrates that a software or systems product is fit for purpose.
That is, it satisfies all the customer's stated an implied needs
Validation can be performed progressively throughout the development life cycle.
Inspection:
For example, written user requirements can be validated by creating a model or prototype and asking the user to confirm (or validate) that the demonstrated functionality meets their needs.
System testing is a major validation event where a system is validated against the user's statement of requirement.
It aims to show that all faults which could degrade system performance have been removed before the system is operated.
Validation is not complete however until the end user formally agrees that the operational system is fit for purpose.
Lecture19

10. 2. Verification & Validation The V Model
Verification - the thing has been built right
Unit testing: Implementation is correct, that is, each bit is correct
Developer test their program code, functions, classes, procedures, to verify that they are all are working as expected.
Bug fixing is less costly and easy in this phase.
Module testing: collection of units, typically a file/class
Integration testing: the bits have been put together correctly
The module, component tested alone should be also work when they connected together.
We need to test the data transfer between the several modules. To ensure that the data transfer between this modules is happening in proper way.
System testing: Architecture/Whole design is correct
The behavior of whole system/product is tested and find out whether system is working as expected or not.
System testing is carried out by specialist’s testers or independent testers.
Lecture19

11. 2. Verification & Validation The V Model (con’t)
Verification - the thing has been built right
Acceptance Test: match the business requirements
The type of testing performed by the Client to check if the system is built to match the business requirements of the organization.
UAT testing is important Because, Developer read the Requirement Document and based on his own understanding developer develop the software which may not actually be what the client needs from the software.
UAT testing happens before releasing the software in a real environment.
Regression testing: Maintenance doesn't break anything that used to work
The type of testing which verifies that software which was previously developed and tested still performs the same way after it was changed or interfaced with other software.
Lecture19

12. 3. Testing Failures vs. Faults vs. Errors
Fault: The abnormal aspect of the code that, if executed, will put the system into an error state
An incorrect step, process, or data definition in a computer program
E.g. forget to initialize a field in one constructor so that it is null when it should not be
Error: The system is in a state different from what it is supposed to be, which if not dealt with, will lead to a failure
E.g. if the constructor is executed, the system is in an error state (but another constructor may be used)
Failure: Externally observed incorrect behavior of code
Incorrect result
E.g. if the field is ever dereferenced, then NullPointerException (but it may never be dereferenced)
Lecture19

13. Return the number of zeros in the array
3. Testing Example
Consider the following code fragment:
Case 1:
Result = 2 -> correct (but error)
Case 2:
Result = 1 -> incorrect (failure)
Return the number of zeros in the array
class IntList {
private int[] values;
...
public int countNumZeros() {
int count = 0;
for (int i = 1; i< values.length; i++)
if (values[i] == 0) count++;
return count; }
int[] input = { 1, 0, 2, 5, 0 }; IntList list = new IntList(input); System.out.println(list.numZeros());
int[] input2 = { 0, 1, 2, 5, 0 };
IntList list2 = new IntList(input2);
System.out.println(list2.countNumZeros());
Lecture19

14. 3. Testing What is Testing?
An activity in which a system or a component is executed under specific conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component.
Testing must be done in an organised, systematic, manner, following a process!
“Program testing can be used to show the presence of bugs, but never to show their absence.” (Dijkstra, 1969)
Proofs of correctness are often as hard (if not harder) to get right than writing the code
Testing vs Debugging:
Testing is (trying) to produce failures
Debugging is (trying to) find the fault that caused failure
Lecture19

15. 3. Testing What is Testing? (con’t)
What kinds of “testing” did you do:
Compiled the program
Entered example values to test your program
Desk-check or code walk through
Inserted System.out.println statements to print out some values
Use debugger (e.g. JBuilder)
Ask your friend to test your program
Lecture19

16. 3.Testing Development Testing
Development testing includes all testing activities that are carried out by the team developing the system.
Basic approaches:
“black-box” – performing tests based on what the component is supposed to do
plug in values & see what happens
Used during all levels of testing
Common approaches:
Equivalence partitioning
Test “boundary” values
“white-box” – performing tests based on how the component is implemented
inspect code/walkthrough code
“Open up” and look at how code works
Select test cases according to program structure (e.g. statement coverage)
Not good at detecting faults relating to requirements
Lecture19

17. 3.Testing Development Testing
Black box or white box
Levels of testing:
Unit testing, where individual program units or object classes are tested. Unit testing should focus on testing the functionality of objects or methods.
Component testing, where several individual units are integrated to create composite components. Component testing should focus on testing component interfaces.
System testing, where some or all of the components in a system are integrated and the system is tested as a whole. System testing should focus on testing component interactions.
Usually black box
Lecture19

18. 4.Unit Testing
Unit testing is the process of testing individual components in isolation.
Units may be:
Individual functions or methods within an object
Object classes with several attributes and methods
Complete test coverage of a class involves
Testing all operations associated with an object
Setting and interrogating all object attributes
Exercising the object in all possible states
Composite components with defined interfaces used to access their functionality.
Lecture19

19. 4.Unit Testing Structure of a Test
Specifies the pre-test state, the inputs, and the expected state or behavior
Test: carry out the test case
Get into the pre-test state
Supply the test inputs (test case values)
Execute with the test case values
Perform any necessary post-test actions to terminate the test
Compare the observed state or behavior with the expected state or behavior
Report results. If the resulting state or behavior matches the expected results, then the test has passed, otherwise it fails
Lecture19

20. Unit Testing Example Customer Maintenance – Black Box
Valid data values are accepted for each data entry field in the interface
Invalid data values entered are identified and appropriate error messages given
Each button responds with a suitable message indicating its action has been carried out
Trans ID
Name
Age
Password
Address Ph
Button
Result 1 10
David 20
ABC
Auckland
Add
Customer 10 added. 2
xxx
Error: Exist 3 12
Mark ab
Error: Age is not a number. 4
Peter
-100
Error: Age is not a valid number. 5
Find
Customer 10 Found. 6 11
Not Found 7
Error: Password - blanked …
Lecture19

21. 4.Unit Testing Test Planning & Documentation
If testing is to be systematic, it must be planned & documented:
how the testing is to be done
who does it
when it is done (schedule)
what level of quality of testing is to be performed
what resources are required
Each individual test case has to be described
how the test results are to be used
There are standards for this kind of thing…
Lecture19

22. IEEE 829-1998 Test Case Specification
Purpose: To define a test case identified by a test design specification.
Outline: A test case specification shall have the following structure:
Test case specification identifier;
Test items;
Input specifications;
Output specifications;
Environmental needs;
Special procedural requirements;
...
Lecture19

23. 4.Unit Testing Test Script
Detailed sequence of steps needed to perform the test
Example: Authenticated Edit of Wiki
Navigate to test page (set up)
Check that Edit tab is visible at top of page (Start)
Create an account or log in link is visible at top right corner (confirms that user is not already authenticated)
Click Edit tab (proceed)
Verify that Login required to edit page shows
Click login link …
This is tedious
Lecture19

24. 4.Unit Testing Automated Testing
Manually running test scripts is tedious, error prone, and expensive
Whenever possible, unit testing should be automated so that tests are run and checked without manual intervention.
In automated unit testing, you make use of a test automation framework (such as JUnit) to write and run your program tests.
Unit testing frameworks provide generic test classes that you extend to create specific test cases. They can then run all of the tests that you have implemented and report, often through some GUI, on the success of otherwise of the tests.
Lecture19