1. Ch. 14 System Testing (with new slides 4/18)
Earlier we have stated the 2 views of testing:
To show that the software works (according to specification)
To detect as many defects (and have them fixed prior to releasing to the users) as possible
In system testing we are more interested in showing that the system, as a whole with all the “major” specifications – especially cross-component, is working.
Still Interested in knowing which “major” function is not working. (But this should be rare if we completed functional and component test properly.)
We are not focused on all types of exception during systems test, but want to make sure that when multiple paths (threads) are executed, it still works according to the requirements.
System testing is a blend of functional (black-box) and structural (white-box) testing testing mostly “main” threads of system behavior.
4/11 PP1 Verification
4/13 PP1 Book says: ?
System Testing

2. 4/13 PP2 Define DD path; DU path; amd MM path. Threads
A thread is a sequence of activities that are observable.
There are many levels of granularity for a thread:
A sequence of instructions as in a DU-path for unit test (DU is Define/Use Testing)
A flow of control among a set of related modules as in a MM-path in an integration path. (MM is module-message or method-message)
A sequence of “atomic” functions that contributes to the accomplishment of a major functionality as described in a requirements specification.
In system testing, we are interested in the
a) high-level (cross-functional) threads and
b) “multiple” threads.
4/13 PP2 Define DD path; DU path; amd MM path.
Threads
A scenario of normal usage A system-level test case A stimulus/response pair Behavior that results from a sequence of system-level inputs An interleaved seq. of port input and output events ……. A seq. of ASFs PP3 Explain in your own words the interleaved seq of… events.
System Testing

3. Examples of threads from the ATM Problem (Section 2.5 p27 of the text)
Unit
Testing
Entry of a digit: (port entry of a numerical digit keystroke and a port out of a digit echo on the screen.
This is a minimal “atomic system function” and qualifies as a thread of interest at the unit test level.
Entry of a PIN number: (this entails a set of activities from screen request of a PIN with interleaved sequence of digit entry until completion possibly including cancellation and up to three re-tries).
This is a sequence of atomic system functions that also qualifies as a thread, probably fit for integration testing or unit testing levels.
A simple ATM transaction: (this includes a set of functions such as 1) Card entry, 2) PIN entry, 3) choice of transaction type (deposit, withdraw, account detail processing, etc.)
This is a sequence of atomic system functions that is a thread & accomplishes a major function, fit for integration testing or system testing levels .
Multiple ATM transactions: (this includes a complete user session where several different transactions- - - query, withdraw, deposit of different accounts may transpire.)
This is a sequence of different/multiple threads and is definitely in the domain of system testing.
System
Testing
System Testing

4. Some definitions related to thread
An atomic system function (ASF) is an action that is observable at the system level in terms of port entry and port output.
An ASF graph is a directed graph of a ASF represented system where the nodes are the ASF’s and the edges represent the sequential flow.
A source ASF is an ASF that is a source node in an ASF graph.
A sink ASF is an ASF that is a sink node in an ASF graph.
A system thread is a path from a source ASF to a sink ASF in the ASF graph of the system.
The thread graph of a system is a directed graph where the
a) nodes are the threads of the system and
b) the edges represent the sequential execution flow from one thread to another.
4/13 PP4 – Explain the source and sink nodes.
System Testing

5. Requirements Specification for System Testing
Requirements specification served as a tool (input) for, mainly black-box testing, but also white-box testing even at the unit testing level; it is even more important as a source of developing test cases for system testing.
There are many ways to express requirements and many ways to view the specification. As a whole the requirements specification may be viewed a set of inter-related elements (“basis” for requirements):
Data (information and information structure)
Actions (functional tasks)
Devices (for input, output, and storage)
Events (triggers and combination of action and data)
Threads (business flow or process flow, involving events)
System Testing
Only thing missing is the non-functional attribute such as quality, performance, etc.

6. Multiple Ways of Viewing Specification “Basis” elements
Used mostly for
early spec and development of system devices
Data
Contextual
model
Actions
Behavior
model
Devices
Often used for requirements
specifications
Events
Structural
model
Threads
Used mostly for
development
There are many tools for expressing these models: (e.g.)
- state transition diagram ( or Finite State Machine)
- E-R diagram
- Petri net
System Testing

7. Using Finite State Machine or State Transition Diagram
Welcome
Card entry
(state A)
State Input Next State Output
A correct card B screen 2
A bad card A screen 1; card
B good pin # C screen 5
B failed pin # A screen 4
wrong card
Display screen 1;
Eject card
Correct card
Display screen 2
Pin
Entry
(state B)
Failed Pin Process
Display screen 4
Successful
Pin Process
Tabular form
Display screen 5
Choosing
Transaction
(state C)
Graphical form
System Testing

8. Finite State Machine at a “deeper” level for PIN processing
Welcome
Card entry
(state A)
Incorrect Pin/
cancel
wrong card
Display screen 4, 1
Display screen 1;
Eject card
Correct card
Display screen 2
1st Pin
Entry
(state B)
Incorrect Pin/
cancel
2nd Pin
Entry
(state D)
Incorrect Pin/
cancel
3rd Pin
Entry
(state E)
Display screen 3, 2
Successful
Pin Process
Display screen 3, 2
Correct Pin
Display screen 5
Choosing
Transaction
(state C)
Display screen 5
Correct Pin
System Testing
Display screen 5

9. 4/18 Details of PIN Entry as a Thread
A screen requesting PIN digits.
An interleaved sequence of digit keystrokes and screen responses.
The possibility of cancellation by the customer before the full PIN is entered.
A system disposition:
A customer has three chances to enter the correct PIN.
Once a correct PIN has been entered, the user sees a screen requesting the transaction type.
After three failed PIN Entry attempts, a screen advises the customer that the ATM card will not be returned, and no access to ATM functions is provided.

10. 4/18 Definition: Atomic System Function
Definition: An Atomic System Function (ASF) is an action that is observable at the system level in terms of port input and output events.
About ASFs
characterized by a sequence of port level inputs and outputs
could be just a simple stimulus/response pair (e.g. digit entry)
Sample ASFs in our ATM example
card entry
PIN entry
Transaction selection
session termination

11. 4/18 More Definitions…
Given a system defined in terms of atomic system functions, the ASF Graph of the system is the directed graph in which nodes are ASFs and edges represent sequential flow.
A source ASF is an Atomic System Function that appears as a source node in the ASF graph of a system.
A sink ASF is an Atomic System Function that appears as a sink node in the ASF graph.
A system thread is a path from a source ASF to a sink ASF in the ASF graph of a system.

12. 4/18 Basis Concepts for Requirements Specification
All of requirements specification models are developed on these basis concepts.
Data
Inputs to actions
Outputs of actions
Events
Actions
Threads (sequences of actions)
Devices

13. 4/18 E/R Model of Basis Concepts

14. 4/18 SATM System User Interface

15. 4/18 SATM System Screens

16. 4/18 Uppermost Level of the SATM Finite State Machine

17. 4/18 Details of SATM PIN Entry State

18. 4/18 Details of SATM PIN Try State

19. 4/18 Paths in the SATM PIN Try State
Correct PIN on first try state sequence
<S2.n.0, S2.n.1, S2.n.2, S2.n.3, S2.n.4, S3>
Port Event Sequence
1st digit, echo “- - - *”
2nd digit, echo “- - * *”
3rd digit, echo “- * * *”
4th digit, echo “* * * *”
Enter
Failed PIN on first try state Sequences
<S2.n.0, S2.n.6>
<S2.n.0, S2.n.1, S2.n.6>
<S2.n.0, S2.n.1, S2.n.2, S2.n.6>
<S2.n.0, S2.n.1, S2.n.2, S2.n.3, S2.n.6>
<S2.n.0, S2.n.1, S2.n.2, S2.n.3, S2.n.4, S2.n.6>

20. 4/18 How Many Paths in the PIN Try State?
1st try: 1 correct + 5 failed attempts
2nd try: 5 failed 1st attempts * 6 second attempts
3rd try: 25 failed 1st and 2nd attempts * six third attempts
Do we really want to test all of these?
This foreshadows the question of “long” versus “short” use cases.

21. 4/18 Port Event Sequence: Correct PIN on 1st Try
Port Input Event
Port Output Event
Screen 2 displayed with ‘ ’
1st digit
Screen 2 displayed with ‘- - - *’
2nd digit
Screen 2 displayed with ‘- - * *’
3rd digit
Screen 2 displayed with ‘- * * *’
4th digit
Screen 2 displayed with ‘* * * *’
(valid PIN)
Screen 5 displayed
4/20 PP – This event sequence would have a Use case ID and description. Why?

22. 4/18 Use Case: Correct PIN on 1st Try
Use Case Name
Correct PIN entry on first try
Use Case ID
EEUC-1
Description
A customer enters the PIN number correctly on the first attempt.
Pre-Conditions
1. The expected PIN is known
2. Screen 2 is displayed
Event Sequence
Input events
Output events
1. Screen 2 shows ' '
2. Customer touches 1st digit
3. Screen 2 shows '- - - * '
4. Customer touches 2nd digit
5. Screen 2 shows '- - * * '
6. Customer touches 3rd digit
7. Screen 2 shows '- * * * '
8. Customer touches 4th digit
9. Screen 2 shows '* * * * '
10. Customer touches Enter
11. Screen 5 is displayed
Post conditions
 Select Transaction screen is active
PP 4/20 – Do you see the information asked about in the last slide?

23. System Test Case: Correct PIN on 1st Try
Test Case Name, ID
Correct PIN entry on first try, TC-1
Description
A customer enters the PIN number correctly on the first attempt.
Pre-Conditions
1. The expected PIN is ‘2468’
2. Screen 2 is displayed
Event Sequence
Input events
Output events
1. Screen 2 shows ' '
2. Customer touches digit 2
3. Screen 2 shows '- - - * '
4. Customer touches digit 4
5. Screen 2 shows '- - * * '
6. Customer touches digit 6
7. Screen 2 shows '- * * * '
8. Customer touches digit 8
9. Screen 2 shows '* * * * '
10. Customer touches Enter
11. Screen 5 is displayed
Post conditions
 Select Transaction screen is active
Test Result, Run by
Pass, Paul Jorgensen

24. 4/18 Event-Driven Petri Net of Correct PIN on First Try
Physical connection require a harness that connects the PC with prototype ports. The test case language consists of the CAUSE and VERIFY verbs, names for port I/O events and device names.

25. 4/18 Automated Test Case Execution
Basic architecture: Use cases interpretively executed
CAUSE inputs
VERIFY outputs
Check observed versus expected outputs
via a harness
connected to the System Under Test

26. 4/18 Executable Test Case for Correct PIN on First Try
Set up requirements: Screen 2 is displayed
CAUSE TouchDigit(2)
VERIFY Screen 2 shows ‘---*’
CAUSE TouchDigit(4)
VERIFY Screen 2 shows ‘--**’
CAUSE TouchDigit(6)
VERIFY Screen 2 shows ‘-***’
CAUSE TouchDigit(8)
VERIFY Screen 2 shows ‘****’
VERIFY Screen 5 is displayed

27. 4/18 Long versus Short Use Cases
A “Long” use case is typically an end-to-end transaction.
SATM example: A full traversal of the high level finite state machine, from the Welcome screen to the End Session screen: <s1, s2, s3, s4, s5>
A “Short” use case is at the level on an atomic system function.
Examples
PIN Entry
Transaction selection
Session closing

28. 4/18 Short Use Cases
A “Short” use case is at the level on an atomic system function (ASF).
In the directed graph of ASFs,
nodes are ASFs
edges signify possible sequential execution of ASFs
Consider an ASF as a “Short” use case, with
pre-conditions
post-conditions
Short use case (ASF) B can follow short use case (ASF) A if the pre-conditions of B are consistent with the post-conditions of A, that is...
Short use cases “connect” at their pre- and post condition boundaries.

29. 4/18 Short Use Cases for the SATM System
Description
SUC1
Valid ATM card swipe
SUC2
Invalid ATM card swipe
SUC3
Correct PIN attempt
SUC4
Failed PIN attempt
SUC5
Choose Balance
SUC6
Choose Deposit
SUC7
Choose Withdrawal: valid withdrawal amount
SUC8
Choose Withdrawal: amount not a multiple of $20
SUC9
Choose Withdrawal: amount greater than account balance
SUC10
Choose Withdrawal: amount greater than daily limit
SUC11
Chose no other transaction
SUC12
Chose another transaction

30. 4/18 Short Use Cases for Failed PIN Attempts
Description
SUC13
Digit 1 entered
SUC14
Digit 2 entered
SUC15
Digit 3 entered
SUC16
Digit 4 entered
SUC17
Enter with valid PIN
SUC18
Cancel before digit 1
SUC19
Cancel after digit 1
SUC20
Cancel after digit 2
SUC21
Cancel after digit 3
SUC22
Cancel after digit 4
SUC23
Enter with invalid PIN
SUC24
Next PIN try
SUC25
Last PIN try

31. 4/18 How Many Use Cases? 1909 “long” use cases 25 “short” use cases
Ways to determine “how many?
Incidence with input events (cover every input event)
Incidence with output events (cover every output event)
Incidence with classes (need a use case/class incidence matrix)
These lead directly to system testing coverage metrics.

32. 4/18 Short Use Cases for the SATM System

33. 4/18 System Testing with Short Use Cases
Basic idea: a short use case is an atomic system function (ASF)
ASFs ...
begin with a port input event
end is one of possibly several port output events
ASFs can be identified
in source code
in executable models
from short use cases
Example: the integration version of NextDate
(see text)

34. Using Operational Profile when interested in “Effectiveness”
The effectiveness of testing is often measured in terms of problems found versus effort spent or
(# of defects found) / person hours spent
(# of defects found) / number of test cases
Historically, most of the failures also tend to fall in small parts of the system (80% failures occur in 20% of the system e.g. the most heavily traversed threads).
Thus if we can collect the operational profiles of the users, we can identify the most heavily used threads.
This strategy of system testing with operational profile can improve our test efficiency.
Not just the number of
problems found

35. Threads with Finite State Machine
If we can model the requirements specifications with the finite state machine model, then:
A set of transitions (with inputs and outputs) can be a source of threads for system test because it covers:
Events ( state and input)
Action (state transition and output)
Data (inputs and outputs)
Thread (series of state transitions)
The drawbacks are:
that not all specifications “easily” fit this model
that the determination of what depth level should the finite state machine be for system test; very low a level is more appropriate for unit or integration test
System Testing

36. Test Coverage Metrics with Finite State Machine
Every node (state) is covered.
This is a very minimal coverage, much like every statement is covered (recall this from structural path testing)
In our 1st state transition chart this is equivalent to covering states A, B, and C with correct card and successful PIN processing.
Every node and edge (transition) is covered
This is a more complete coverage if the finite state machine is carried to deep enough level.
System Testing

37. Strategies for Thread Testing Finite State Machine modeled systems
Event based (state-input based):
Each input event occurs
A “typical” or “common” sequence of input events occur
Each input event occurs in every “relevant” data context
For a given context, all “inappropriate” input events occur
For a given context, all possible (both appropriate & inappropriate) inputs events occur
Event based (state-output based):
Each output event occurs
Each output event occurs for every “cause”
Port-based (both input and output):
Cover all input and out ports
System Testing

38. Strategies for thread Testing Data Represented Systems
Some software system requirements are better represented with E-R diagram.
For E-R modeling used for specifications, the threads may be identified via:
Looking at the cardinality of every relationship
One-one
One-many
Many-one
Many - many
Looking at the participation factor of every relationship
Looking at functional dependencies among relationships
e.g. loaning a book that’s not available
More importantly --- we want to make sure that the DB “access” is tested,
along with threads such as a) shutdown -restart b) backup-recovery
c) multiple access threads
System Testing

39. Using Operational Profile when interested in “Effectiveness”
The effectiveness of testing is often measured in terms of problems found versus effort spent or
(# of defects found) / person hours spent
(# of defects found) / number of test cases
Historically, most of the failures also tend to fall in small parts of the system (80% failures occur in 20% of the system e.g. the most heavily traversed threads).
Thus if we can collect the operational profiles of the users, we can identify the most heavily used threads.
This strategy of system testing with operational profile can improve our test efficiency.
Not just the number of
problems found
System Testing

40. User-Operational Profile Example (naïve user versus experienced user)
(20/20)
(20/20)
prob = .25 √
term
naïve users
(20/80)
default f1 f4
(5/5)
prob = .063
(5/10) f4
term
Start
(80 users)
(5/10)
(5/5)
prob = .063
(10/60) f1 f6
term
experienced
users (60/80)
(40/40)
(40/40)
(40/60)
prob = .50√√
term f7 f2
Choice of
f1,f2,f3
(10/60)
(3/10)
(3/3)
prob = .037
term f4 f3
(7/10)
(7/7)
prob = .087 f5
Note that 2 usage cases take
up 75% of operational profile !
term
System Testing

41. Software changes and Regression Test
In the life span of a software, we can anticipate changes that result in multiple cycles of fixes and release. Thus it needs to be retested to ensure that what worked before still works (did not get regressed).
A common strategy is to re-run the complete system test over as the regression test
A more economical strategy for regression test is to develop (1) new threads for the new areas and (2) only include those old threads that are affected by the new release. (Yes, there may be an element of risk)
Think about the
“neighborhood”
testing strategy-last lecture
System Testing