1. jpf@fe.up.pt www.fe.up.pt/~jpf
TQS - Teste e Qualidade de Software (Software Testing and Quality) Test Case Design – Black Box Testing
João Pascoal Faria

2. Preâmbulo – Bugs recentes (1)
tirando EUR funciona!

3. Preâmbulo – Bugs recentes (2)
Excel 2003 (certo)
Excel 2007 (errado)
Problemas evitáveis com boas técnicas de concepção de casos de teste?

4. Index Equivalence class (or domain) partitioning
Boundary-value analysis
Cause-effect graphing (or decision table)
Error guessing
Risk based testing
Testing for race conditions
Random testing
Requirements based testing

5. Equivalence class (or domain) partitioning
Partir domínio de valores de entrada (ou saída) em classes com comportamento esperado similar, e testar pelo menos um valor de cada classe
Classes podem ser formalizadas por condições
Exemplo a testar abs(x) : x < 0, x >= 0
Considerar classes de valores válidos e valores inválidos
Exemplo a testar sqrt(x) : x < 0 (inválido), x >= 0 (válido)
A partição pode ser multi-critério (multi-dimensional)
Exemplo a testar pesquisa de string num texto:
Quanto ao resultado: existe, não existe
Quanto à distinção de maiúsculas/minúsculas: distingue, não distingue
Um caso de teste pode cobrir várias classes de valores válidos, mas não deve cobrir várias classes de valores inválidos

6. Equivalence class partitioning - Example 1
Test a function that calculates the absolute value of an integer x
Equivalence classes :
Criteria Valid eq. classes Invalid eq. classes
nr of inputs , > 1
input type integer non-integer
particular x < , >= 0
(1) (2) (3)
(4) (5)
(6) (7)
Test cases :
x = (1,4,6)
x = (1,4,7)
x = (2)
x = (3)
x = “XYZ” (5)

7. Boundary value analysis
Testar valores na fronteira de cada classe e imediatamente acima e abaixo (além de ou em vez de valores intermédios)
Testar também valores especiais (null, 0, etc.)
Exemplo a testar abs(x):
Classe x <0 : x = min, x = -1
Classe x >= 0 : x = 0, x = 1, x = max
Baseado na observação de que bugs ocorrem frequentemente em valores fronteira
Problemas com índices de arrays, decisões, overflow, etc.
Se o sistema se comportar bem nos casos fronteira então provavelmente comportar-se-á bem em valores intermédios
“Bugs lurk in corners and congregate at boundaries.” (B.Beizer)

8. Boundary value analysis Example 3: search routine specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : out BOOLEAN; L: out ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
(source: Ian Sommerville)

9. Boundary value analysis Example 3 - input partitions
P1 - Inputs which conform to the pre-conditions (valid)
array with 1 value (boundary)
array with more than one value (different size from test case to test case)
P2 - Inputs where a pre-condition does not hold (invalid)
array with zero length
P3 - Inputs where the key element is a member of the array
first, last and middle positions in different test cases
P4 - Inputs where the key element is not a member of the array

10. Boundary value analysis Example 3 – test cases (valid cases only)

11. A self-assessment test [Myers]
“A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene (all lengths are different), isosceles (two lengths are equal), or equilateral (all lengths are equal).”
Write a set of test cases to test this program.
Inputs: l1, l2, l3 , integer, li > 0, li < lj + lk
Output: error, scalene, isosceles or equilateral

12. A self-assessment test [Myers]
Test cases for:
valid inputs:
invalid inputs:
valid scalene triangle ?
valid equilateral triangle ?
valid isosceles triangle ?
3 permutations of previous ?
side = 0 ?
negative side ?
one side is sum of others ?
3 permutations of previous ?
one side larger than sum of others ?
all sides = 0 ?
non-integer input ?
wrong number of values ?

13. Cause-effect graphing
Black-box technique to analyze combinations of input conditions
Identify causes and effects in specification   inputs / outputs / initial state final state conditions conditions
Make Boolean Graph linking causes and effects
Annotate impossible combinations of causes and effects
Develop decision table from graph with a particular combination of inputs and outputs in each column
Transform each column into a test case

14. Example Requirements for Calculating Car Insurance Premiums:
R1 For females less than 65 years of age, the premium is $500
R2 For males less than 25 years of age, the premium is $3000
R3 For males between 25 and 64 years of age, the premium is $1000
R4 For anyone 65 years of age or more, the premium is $1500

15. Example – step 1
Identify the causes and effects.

16. Example – step 2
The semantic content of the specification is analyzed and transformed into a Boolean graph linking the causes and effects. This is the cause-effect graph.

17. Example – step 2 (cont.)

18. Example – step 3
Annotate the graph with constraints describing combinations of causes and/or effects that are impossible because of syntactic or environmental constraints or considerations.
The “O” (One and Only One) constraint states
that one and only one of the causes 1 and 2
can be true.

19. Example – step 4
Methodically trace state conditions in the graphs, converting them into a limited-entry decision table. Each column in the table represents a test case.

20. Example – step 5
The columns in the decision table are converted into test cases.

21. Cause-effect graphing
Systematic method for generating test cases representing combinations of conditions
Differently from eq. class partitioning, we define a test case for each possible combination of conditions
Drawback: Combinatorial explosion of number of possible combinations

22. Error guessing Just ‘guess’ where the errors are ……
Intuition and experience of tester
Ad hoc, not really a technique
But can be quite effective
Strategy:
Make a list of possible errors or error-prone situations (often related to boundary conditions)
Write test cases based on this list

23. Risk based testing More sophisticated ‘error guessing’
Try to identify critical parts of program ( high risk code sections ):
parts with unclear specifications
developed by junior programmer while his wife was pregnant ……
complex code (white box?): measure code complexity - tools available (McGabe, Logiscope,…)
High-risk code will be more thoroughly tested ( or be rewritten immediately ……)

24. Testing for race conditions
Also called bad timing and concurrency problems
Problems that occur in multitasking systems (with multiple threads or processes)
A kind of boundary analysis related to the dynamic views of a system (state-transition view and process-communication view)
Examples of situations that may expose race conditions:
problems with shared resources:
saving and loading the same document at the same time with different programs
sharing the same printer, communications port or other peripheral
using different programs (or instances of a program) to simultaneously access a common database
problems with interruptions:
pressing keys or sending mouse clicks while the software is loading or changing states
other problems:
shutting down or starting two or more instances of the software at the same time
Knowledge used: dynamic models (state-transition models, process models)
(source: Ron Patton)

25. Random testing Input values are (pseudo) randomly generated
(-) Need some automatic way to check the outputs (for functional / correctness testing)
By comparing the actual output with the output produced by a "trusted" implementation
By checking the result with some kind of procedure or expression
Some times it's much easier to check a result than to actually computing it
Example: sorting - O(n log n) to perform; O(n) to check (final array is sorted and has same elements than initial array)
(+) Many test cases may be generated
(+) Repeatable (*): pseudo-random generators produce the same sequence of values when started with the same initial value
(*) essential to check if a bug was corrected
(-) May not cover special cases that are discovered by "manual" techniques
Combine with "manual" generation of test cases for boundary values
(+) Particularly adequate for performance testing (it's not necessary to check the correctness of outputs)

26. Requirements based testing
You have a list (or tree) of requirements (or features or properties)
Define at least one test case for each requirement
Build and maintain a (tests to requirements) traceability matrix
Particularly adequate for system and acceptance testing, but applicable in other situations
Hints:
Write test cases as specifications by examples
Write distinctive test cases (examples)
Not enough: Requirement1 or Requirement2 => TestCase1Behaviour
Test separately and in combination
Write “positive” (examples in favor) and “negative” (examples against) test cases

27. Black box testing: which One ?
Black box testing techniques :
Equivalence class partitioning
Boundary value analysis
Cause-effect graphing
Error guessing
…………
Which one to use ?
None of them is complete
All are based on some kind of heuristics
They are complementary

28. Black box testing: which one ?
Always use a combination of techniques
When a formal specification is available try to use it
Identify valid and invalid input equivalence classes
Identify output equivalence classes
Apply boundary value analysis on valid equivalence classes
Guess about possible errors
Cause-effect graphing for linking inputs and outputs

29. References and further reading
Practical Software Testing, Ilene Burnstein, Springer-Verlag, 2003
Software Testing, Ron Patton, SAMS, 2001
The Art of Software Testing, Glenford J. Myers, Wiley & Sons, (Chapter 4 - Test Case Design)
Classical
Software testing techniques, Boris Beizer, Van Nostrand Reinhold, 2nd Ed, 1990
Bible
Testing Computer Software,  2nd Edition, Cem Kaner, Jack Falk, Hung Nguyen, John Wiley & Sons, 1999
Practical, black box only