1. System Testing

2. Overview System Testing General - Introduction Threads
Basis Concepts for Requirements Specification
Finding Threads
Structural Strategies for Thread Testing
Functional Strategies for Thread Testing
System Testing Guidelines
Ref: “Software Testing A Craftsman's Approach” 2nd edition, Paul C. Jorgensen

3. System Testing Threads -Threads are hard to define
several views of a thread:
•  a scenario of normal usage
•  a system level test case
•  a stimulus/response pair
•  behavior that results from a sequence of system level inputs
•  an interleaved sequence of port input and output events
•  a sequence of transitions in a state machine description of the system
•  an interleaved sequence of object messages and method executions
•  a sequence of machine instructions
•  a sequence of source instructions
•  a sequence of atomic system functions

4. Threads have distinct levels.
A unit level thread - an execution-time path of source instructions, or alternatively as a path of DD -Paths.
An integration level thread is a sequence of MM-Paths (msg/method)that implements an atomic system function.
A system level thread is a sequence of atomic system functions.
Unit testing tests individual functions,
Integration testing examines interactions among units, and
System testing examines interactions among atomic system functions.

5. Thread Possibilities
three candidate threads:
Entry of a digit
Entry of a Personal Identification Number (PIN)
A simple transaction: ATM Card Entry, PIN entry, select transaction type (deposit, withdraw), present account details (checking or savings, amount), conduct the operation, and report the results.
An ATM session, containing two or more simple transactions.

6. Digit entry is a good example of a minimal atomic system function that is implemented with a single MM-Path.
It begins with a port input event (the digit keystroke) and ends with a port output event (the screen digit echo), so it qualifies as a stimulus/response pair.
Second candidate, PIN Entry, is a good example of an upper limit to integration testing, and at the same time, a starting point of system testing.
PIN Entry is a good example of an atomic system function.
It is also a good example of a family of stimulus/response pairs (system level behavior that is initiated by a port input event, traverses some programmed logic, and terminates in one of several possible responses [port output events]).

7. The third candidate, the simple transaction, has a sense of “end-to-end” completion.
A customer could never execute PIN Entry all by itself (a Card Entry is needed), but the simple transaction is commonly executed.
This is a good example of a system level thread; note that it involves the interaction of several ASFs. (Atomic S/m Fn.,)
The last possibility (the session) is really a sequence of threads.
This is also properly a part of system testing; at this level, we are interested in the interactions among threads.
Unfortunately, most system testing efforts never reach the level of thread interaction.

8. Thread Definitions
A unit thread is a path in the program graph of a unit.
There are two levels of threads used in integration testing: MM-Paths and atomic system functions
Recall that MM-Paths are defined as paths in the directed graph in which
module execution paths are nodes, and
edges show execution time sequence.
An MM-Path is a path in the MM-Path graph of a set of units.
Given a system defined in terms of atomic system functions, the ASF Graph of the system is the directed graph in which
nodes are atomic system functions and
edges represent sequential flow.

9. A source ASF is an atomic system function that appears as a source node in the ASF graph of a system; similarly, a sink ASF is an atomic system function that appears as a sink node in the ASF graph.
In the SATM system, the Card Entry ASF is a source ASF, and the session termination ASF is a sink ASF. Notice that intermediary ASFs could never be tested at the system level by themselves — they need the predecessor ASFs to “get there”.
A system thread is a path from a source ASF to a sink ASF in the ASF graph of a system.
Given a system defined in terms of system threads, the Thread Graph of the system is the directed graph in which
nodes are system threads and
edges represent sequential execution of individual threads.

10. Basis Concepts for Requirements Specification
Data
Actions
Ports
Events
Threads

11. Finding Threads A hierarchy of state machines;
the upper level - states correspond to stages of processing, and transitions are caused by logical (rather than port) events.
The Card Entry “state” for example, would be decomposed into lower levels that deal with details like jammed cards, cards that are upside-down, stuck card rollers, and checking the card against the list of cards for which service is offered.

12. Structural Strategies for Thread Testing
Bottom-up Threads
When we organize state machines in a hierarchy, we can work from the bottom up.
There are six paths in the PIN Try state machine.
If we traverse these six, we test for three things:
correct recognition and echo of entered digits,
response to the cancel keystroke, and
matching expected and entered PINs.

13. Node and Edge Coverage Metrics
Node (state) coverage is analogous to statement coverage at the unit level — it is the bare minimum.
In the PIN Entry example, we can attain node coverage without ever executing a thread with a correct PIN.
Edge (state transition) coverage is a more acceptable standard.
If the state machines are “well formed” (transitions in terms of port events), edge coverage also guarantees port event coverage.

14. Functional Strategies for Thread Testing
If no behavioral model [i.e. FSMs] exists for a system,
then two choices:
develop a behavioral model or
resort to system level analogs of functional testing.
To identify functional test cases, we used information from the input and output domains [spaces] as well as the function itself.
Functional threads are described in terms of coverage metrics that are derived from:
events, ports, and data.

15. Event Based Thread Testing
Considering the space of port input events, the following port input thread coverage metrics are of interest to attain levels of system testing:
PI1: each port input event occurs
PI2: common sequences of port input events occur
PI3: each port input event occurs in every “relevant” data context
PI4: for a given context, all “inappropriate” input events occur
PI5: for a given context, all possible input events occur

16. PI1 is bare minimum and adequate for most systems.
PI2 is the most common and corresponds to the intuitive view of the system since it deal with normal or expected use - hard to quantify.
a view of a ‘context’ [for PI3 - PI5] is that of event quiescence (inactive).
For example, PI3 deals with context sensitive port input events
physical events that have logical meanings determined by the context within which they occur.
PI3 is driven by an event in all of its contexts.

17. PI4 and PI5 both start with a context and then seek a variety of events.
PI4 is often used by a tester in an attempt to “break” a system - supply inappropriate inputs just to see what happens
This is partially a specification problem
differences between prescribed behavior [things that should happen] and proscribed behavior [things that should not happen].
Most requirement specs have difficulty with prescribed behavior –
usually testers find proscribed behavior.

18. Coverage Metrics for Port Output Events
PO1: each port output event occurs.
PO1 is acceptable minimum especially when a system has a rich variety of output messages for error conditions.
PO2: each port output event occurs for each cause.
PO2 is good but hard to quantify
Basically it refers to threads that interact with respect to a port output event.
Usually a given port output event has a small number of causes.

19. Port-Based Thread Testing
Complements event-based testing:
Determine, for each port, what events can occur at that port;
Find threads which exercise input ports and output ports with respect to the event list for each port.
not all systems will have this characteristic
i.e. an event that occurs at more than one port
Useful for systems in which the port devices come from external suppliers.
From an ER diagram, any N:N relationships between ports and events should be exercised in both directions.
Event based testing covers the 1:N relationship from events to ports,
Port based testing covers the 1:N relationship from ports to events.

20. Data-Based Thread Testing
Port and event based testing work best in event driven (reactive) systems.
Reactive systems are often characterized as: long running and maintain a relationship with their environment.
Very seldom do they have an interesting data-model so data threads are not very useful.
Non-reactive systems are typically “static” and are transformational (rather than reactive):
essentially support operations on a ‘database’ and where the ER model is dominant.

21. Define sets of threads in terms of data-based coverage metrics.
Information in relationships has many system threads, whereas the threads in the entities are usually handled at the unit level.
We can define the following metrics:
DM1: exercise the cardinality of every relationship .
DM2: exercise the participation of every relationship .
DM3: exercise the functional dependencies among relationships.

22. DM1:
Refers to the 4 possibilities of rlnship., - one to one, one to many, many to one, many to many.
DM2:
Refers to whether every instance of an entity participates in a relationship.
Some modeling techniques express participation in numerical limits (i.e. OMT, “at least one and at most 12”).
When available, this information leads to boundary value system threads.
DM3:
Transactions determine explicit logical connections among relationships known as functional dependencies
These are reduced when the database is normalized, but they still exist and lead to interesting s/m test threads.

23. SATM Test Threads
SATM system, we get a set of threads that constitutes a thorough system level test.
We develop such a set of threads here in terms of an overall state model in which states correspond to key atomic system functions.
The macro-level states are:
Card Entry, PIN Entry,
Transaction Request, (and processing), and
Session Management.

24. screen 15, eject card, screen 1
SATM Test Data
PAN
Expected PIN
Checking Balance
Savings Balance
100
1234 $
$800.00
200
4567
$100.00
$90.00
300
6789
$25.00
$20.00
Thread 1 (balance)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
1234
B1, B1 B2
Port Outputs
screen 2
screen 5
screen 6, screen 14 $
screen 15, eject card, screen 1

25. screen 6, screen 7, screen 13, dep. door opens, screen 14, $1025.00
Thread 2 (deposit)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
1234
B2, B1, insert env. B2
Port Outputs
screen 2
screen 5
screen 6, screen 7, screen 13, dep. door opens, screen 14, $
screen 15, eject card, screen 1
Thread 3 (withdrawal)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
1234
B3, B2, 30.00 B2
Port Outputs
screen 2
screen 5
screen 6, screen 7, screen 11, withdrawal door opens, 3 $10 notes, screen 14, $770.00
screen 15, eject card, screen 1
Thread 4
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
400
Port Outputs
eject card screen 1

26. Thread 5 (balance)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
as in thread 1
Port Outputs
screen 2
screens 3,2,5
Thread 6 (balance)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
C1234
as in thread 1
Port Outputs
screen 2
screens 3,2,5
Thread 7 (balance)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
1C12C1234
as in thread 1
Port Outputs
screen 2
screens 3,2, 3,2,5
Thread 8 (balance)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
123C1C1C
Port Outputs
screen 2
screens 3,2, 3,2,4,1

27. screen 15, eject card, screen 1
Thread 9 (withdrawal)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
1234
B3, B2, Cancel B2
Port Outputs
screen 2
screen 5
screens 6,7, 9, 7
screen 15, eject card, screen 1
Thread 10 (withdrawal)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
300
6789
B3, B2, Cancel B2
Port Outputs
screen 2
screen 5
screens 6,7,8
screen 15, eject card, screen 1
Thread 11 (withdrawal)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
1234
B3, B2, Cancel B2
Port Outputs
screen 2
screen 5
screens 6,7, 10
screen 15, eject card, screen 1
Thread 12 (balance)
Card Entry (PAN)
PIN Entry
Transaction Request
Session Management
Port Inputs
100
1234
B1, B1
B1, Cancel
Port Outputs
screen 2
screen 5
screen 6, screen 14 $
screen 15, screen 5, screen 15, eject card, screen 1

28. Threads for Context Sensitive Input Events
Keystroke
Screen
Logical Meaning 6
cancel 2
PIN Entry error 14 5
transaction selection error 15
account selection error 16 7
amount selection error 17 8 18 13
deposit envelope not ready 1 B1
balance
checking 19 10
yes (a non-withdrawal transaction) 20 12
yes (a non-deposit transaction)
yes (another transaction) B2
deposit 3
savings 21
no (no additional transaction) 22

29. System Testing Guidelines
Pseudo-Structural System Testing
At the system level we can use graph based metrics as a cross check on the functional threads.
The claim is for pseudo-structural testing since the node and edge coverage metrics are defined in terms of a control model of a system and not directly derived from the system implementation.
Behavioral models are only an approximation to the system’s reality:
why it is possible to decompose the models to several levels of detail.
A true structural model’s sheer size and complexity would make it too cumbersome to use.

30. The weakness of pseudo-structural metrics is that the underlying model may be a poor choice.
Decision tables and FSMs are good choices for ASF testing. For example:
If an ASF is described using a decision table, conditions usually include port input events, and actions are port output events
It is possible to devise test cases that cover every condition, every action, or most completely, every rule.
As in FSMs, test cases can cover every state, transition or every path.

31. Operational Profiles
Many aspects of testing can be related to the old 80/20 rule:
for a system with many threads, 80% of the execution traverses only 20% of the threads.
The basic concept in testing:
execute test cases such that when a failure occurs, the presence of a fault is revealed.
The distribution of faults is only indirectly related to the reliability of the system.
Simple view of reliability:
The probability that no failures occur during a specific time interval.
If faults are in less traveled threads of system,
then reliability will appear higher than if the same number of faults were in the high traffic areas.
Operational profiles:
Determine the execution frequencies of various threads, then select threads accordingly.
Operational profiles maximize the probability of finding faults by inducing failures in the most frequently traversed threads.

32. Progression vs. Regression Testing
The most common approach to regression testing is to simply repeat the system tests.
We can refine this (and drastically reduce the effort) by choosing test threads with respect to the goals of regression and progression testing.
With progression testing, we are testing “new territory”,
so we expect a higher failure rate than with regression testing.
Another difference: because we expect to find more faults with progression testing, we need to be able to locate the faults. This requires test cases with a “diagnostic” capability, that is, tests that can fail only a few ways.
For thread based testing, progression testing should use shorter threads that can fail only in a few ways.
These threads might be ordered as we did with the SATM thread test set, such that longer threads are built up from shorter (and previously tested) threads.

33. We have lower expectations of failure with regression testing, and we are less concerned with fault isolation.
Taken together, this means regression testing should use longer threads that can fail in several ways.
If we think in terms of coverage, both progression and regression testing will have thorough coverage, but the density is different.
State and transition coverage matrices (like Tables 8 and 9) will be sparse for progression testing threads and dense for regression testing threads.
As a rule, “good”
regression testing threads will have low operational frequencies, and
progression testing threads will have high operational frequencies.

34. System Testing Categories
Objective of system testing:
To verify whether the implementation (or system) conforms to the requirements as specified by the customer(s).
To verify whether the system meets a wide range of unspecified expectations
System testing is performed
After constructing a reasonably stable system in an emulated environment.
In the real environment (if the real environment is not accessible, system testing is done using models in an emulated environment)

35. Testing categories (11 categories)
It is important to categorize the kinds of system tests for a number of reasons:
Systematically focus on different aspects of a system while evaluating its quality.
Test engineers can prioritize their activities.
Planning based on test categorization has the advantage of obtaining a balanced view of testing.
Testing categories (11 categories)

36. Basic System Tests
These provide evidence that the system can be installed, configured and brought to an operational state.
Basic tests are performed to ensure that commonly used functions, not all of which may directly relate to user-level functions, work to our satisfaction.
The following are the major categories of subsystems whose adequate testing is called basic test.
a. Boot tests: verify that the system can boot up its software image from the supported options (ROM, PCMCIA, ...)
b. Upgrade/downgrade tests: verify that the software image can be upgraded or downgraded (rolled back) in a graceful manner.

37. c. Light emitting diode (LED) tests
c. Light emitting diode (LED) tests. These are designed to ensure that that visual operational status of the system is correct.
d Diagnostic tests. These are designed to ensure that the hardware components of the systems are functioning as desired. (Power-on self test (POST), Memory, Address and data buses, Peripheral devices)
e) CLI (command line interface) tests. Ensure that the system can be configured. Ensure that uses commands are properly interpreted. Verify the error message

38. Functionality Tests
Verify the system as thoroughly as possible over the full range of requirements
Logging and tracing tests: (implicit functionality)
GUI tests: (Icon, Menu bar, Dialog box, Scroll bar)
Security tests. Verify that the system meets the requirements for detecting security breaches and protecting from such breaches (Unauthorized access, Illegal file access, Virus)

39. Robustness Tests
To verify how gracefully the system behaves in error situations, or how it handles a change in its operational environment.
Different kinds of tests are:
Boundary value tests (valid and invalid inputs)
Recover from power failure
On-line insertion and removal (OIR)
System recovers after an OIR event
Availability of redundant modules
Degraded node test (a portion of the system fails)

40. Interoperability and Performance Tests
Interoperability tests (3rd party products): Verify that the system can inter operate with 3rd party products.
Performance tests: Determine how actual system performance compares to predicted performance
Response time
Execution time
Throughput
Resource utilization
Traffic volume

41. Scalability Tests
Verify that the system can scale up to its engineering limits
Data storage limitations (counters and buffers)
Speed limitations (CPU)
Communication limitations
Resource intensive

42. Stress Tests
Stress tests (push it over the edge to break it). Evaluate the behavior of a software component when offered load is in excess of its designed capacity
Push the system “over the edge” and observe that the recovery mechanism works
Bring out the following kinds of problems:
Memory leaks
Buffer allocation problem

43. Load and Regression Tests
Load and stability tests: Verify that the system can operate in a large scale for a long time (months)
Regression tests: Ensure that nothing had has happened after a fix.
Five possibilities can happen after an attempt to fix.
fix the bug reported
fail to fix the bug
fix the bug, but break something else
fail to fix the bug, but break something else
fix this bug and fix some unknown bugs

44. Documentation Tests
Documentation tests: This is a review of technical accuracy and readability of user manuals including
tutorials
on-line help
There are three kinds of documentation tests
Read test (clarity, organization, flow and accuracy)
Hands-on test (evaluate usefulness)
Functional test (verify the document)

45. Conformance to Regulatory Bodies
Identify unsafe consequences
Federal Communication Commission (FCC) and Canadian Standards
Association (CSA) certify a product's safety.