1. Black Box Testing (Chapter 4)

2. 6/23/2018
Black Box Testing

3. Objectives To understand: What black box testing is
The differences between black box testing and white box testing
Different techniques of black box testing
6/23/2018
Black Box Testing

4. What is Black Box Testing?
Black box testing is done without the knowledge of the internals of the system under test.
6/23/2018
Black Box Testing

5. Characteristics of Black Box Testing
Done based on requirements
Addresses (should address) stated as well as implied requirements
Encompasses the end-user perspective
Checks for valid and invalid conditions / inputs
May or may not know the technology aspects of the product
6/23/2018
Black Box Testing

6. Differences Between White Box and Black Box Testing
White Box Testing
Has no access to program code
Has access to program code
Requires external perspective
Requires knowledge of program code
Set of techniques applicable to all other phases of testing
Typically applies only to unit testing, where code is involved
6/23/2018
Black Box Testing

7. Principles of Black Box Testing
It is not possible to exhaustively test a product
Choose tests so that we can
Test “as much of the external functionality as possible”
Uncover “as many defects as possible”
Use “as short a time as possible”
Have methodologies that choose tests that have
a higher likelihood of uncovering new defects.
6/23/2018
Black Box Testing

8. Lock-and-Key Example of Black Box Testing
6/23/2018
Black Box Testing

9. 6/23/2018
Black Box Testing

10. 6/23/2018
Black Box Testing

11. Techniques / Methodologies of Black Box Testing
Requirements-based testing
Positive and negative testing
Boundary value analysis
Decision tables
Equivalence partitioning
State-based testing
Compatibility testing
User documentation testing
Domain testing (leads to ad hoc testing)
6/23/2018
Black Box Testing

12. General Format of Discussion of Techniques
Present some reasoning where applicable.
List out one or two examples.
Walk through the examples.
Summarize the process for using the technique.
6/23/2018
Black Box Testing

13. Requirements-Based Testing
Done to ensure that all requirements in SRS are tested
Difference between implicit and explicit requirements
Review requirements first to ensure they are consistent, correct, complete and testable
Review enables translation of (some of) the implied requirements to stated requirements
A reviewed SRS tabulates requirements, along with a requirements id and a priority
This is the genesis of a Requirements Traceability Matrix
6/23/2018
Black Box Testing

14. Requirements Traceability Matrix – General Format
Refer to Table 4.2, page 78.
6/23/2018
Black Box Testing

15. 6/23/2018
Black Box Testing

16. 6/23/2018
Black Box Testing

17. Zooming-In on the Testing Part of RTM
Table 4.2
Test condition: Different ways of testing a requirement (different types of mapping)
Test case: Different conditions / scenarios for a given requirement
Phase of testing – helps scheduling
Importance of documenting expected results
6/23/2018
Black Box Testing

18. Positive and Negative Testing
Positive testing to check that the product does what it is supposed to
Behaves correctly when given right inputs
Maps to a specific requirement
“Coverage” is defined better
Negative testing to show that the product does not fail when given unexpected inputs
Tries to break the system
No direct mapping to a specific requirement
“Coverage” more challenging
Table 4.4 and 4.5, Page 83
6/23/2018
Black Box Testing

19. 6/23/2018
Black Box Testing

20. Boundary Value Analysis
Most defects come up near “boundaries”
Reasons from a white box perspective:
Programmers tentativeness in using the right relational operator (< or < = ?)
Multiple ways of implementing loops
Confusing array subscripts
Reasons from a black box perspective:
Unclear requirements
Ambiguous or “it depends” mindset!
6/23/2018
Black Box Testing

21. Boundary Conditions – Example 1: Database Buffers
Database starts with a pre-allocating number of buffers for caching
Buffers filled up as needed
If full, buffers freed on a FIFO basis
6/23/2018
Black Box Testing

22. 6/23/2018
Black Box Testing

23. Example 2: Volume Discount Calculations
Product pricing depends on the quantity procured
Based on pre-defined “slabs”
Discussion of example in Pages 84/85
6/23/2018
Black Box Testing

24. 6/23/2018
Black Box Testing

25. Boundary Value Analysis (BVA): Summary of Process
Look for any kind of gradation or discontinuity in data values that affects computation – the discontinuities are the boundary values, requiring thorough testing.
Look for any internal limits like limits on resources (like the example of buffers given above). The behaviour of the product at these limits should also be the subject of boundary value testing.
Also include in the list of boundary values, documented limits on hardware resources. For example, if it is documented that a product will run with minimum 4MB of RAM, make sure you include test cases for the minimum RAM (i.e., 4MB in this case).
The examples given above discuss boundary conditions for input data – the same analysis needs to be done for output variables also.
6/23/2018
Black Box Testing

26. Decision Tables
A program’s behaviour is characterized by several decision variables.
Each decision variable specifies a boolean condition.
The distinct combinations of these decision variables lead to different scenarios.
Each scenario occupies a row in the decision table, and the row also has expected results.
One representative data point from each scenario needs to be tested.
6/23/2018
Black Box Testing

27. Example of Decision Tables: Pages 87-90
6/23/2018
Black Box Testing

28. 6/23/2018
Black Box Testing

29. 6/23/2018
Black Box Testing

30. Summary of Decision Table
Identify the decision variables.
Identify the possible values of each of the decision variables.
Enumerate the combinations of the allowed values of each of the variables.
Identify the cases in which values assumed by a variable (or by sets of variables) are immaterial for a given combination of other input variables. Represent such variables by the Don’t Care symbol.
For each combination of values of decision variables (appropriately minimized with the Don’t Care scenarios), list out the action or expected result.
Form a table, listing on each but the last column one decision variable. On the last column, list the action item for the combination of variables in that row (including Don’t Cares, as appropriate)
6/23/2018
Black Box Testing

31. Equivalence Partitioning (EVP)
Generalization of BVA / decision table
Divide the (potentially infinite) set of values into a set of equivalence classes or partitions
One element of the class can act as a representative for the entire class
Results of the test for that one element is extrapolated to all the other elements
6/23/2018
Black Box Testing

32. Example of EVP
Pages 91-93
6/23/2018
Black Box Testing

33. 6/23/2018
Black Box Testing

34. Summary of Process for EVP
Choose criteria for doing the equivalence partitioning (range, list of values, etc.).
Identify the valid equivalence classes based on the above criteria (number of ranges, allowed values, etc.) .
Select a sample data from that partition.
Write the expected result based on the requirements given.
Identify special values, if any, and include them in the table.
Check to have expected results for all the cases prepared
If the expected result is not clear for any particular test case, mark appropriately and escalate for corrective actions. If you cannot answer a question, or find an inappropriate answer, consider whether you want to record this issue on your log.
6/23/2018
Black Box Testing

35. State- or Graph-Based Testing
Useful for
Language processors or compilers
Workflow modelling
Dataflow modelling
6/23/2018
Black Box Testing

36. Example of Graph-Based Testing for an SQL Command1
SELECT 2 * 3
<col name> 4 , 5
FROM 6
<table name> 7
WHERE …
6/23/2018
Black Box Testing

37. 6/23/2018
Black Box Testing

38. Process for Language processors Testing Using Graph Method
Identify the grammar for the scenario. In the above example, we have represented the diagram as a state machine. In some cases, the scenario can be a context-free grammar, which may require a more sophisticated representation of a “state diagram”.
Design test cases corresponding to each valid state-input combination.
Design test cases corresponding to the most common invalid combinations of state-input.
6/23/2018
Black Box Testing

39. Requestor Approving Authority Finance Vendor Received Item
Available
Off the shelf
Buy &
Expense
(Transaction:
Expense Report)
Requisition
Approval
Budget Update &
Purchase Order
Delivery
Note
Payment
Item Delivery
6/23/2018
Black Box Testing

40. Workflow Modelling
6/23/2018
Black Box Testing

41. General Characteristics of Graph-Based Testing
The application can be characterized by a set of states.
The data values (screens, mouse clicks, etc.) that cause the transition from one state to another are well understood.
The methods of processing within each state to process the input received are also well understood.
6/23/2018
Black Box Testing

42. Compatibility Testing
To ensure that the product works consistently with infrastructure components
Could be parameters of hardware, OS, network…
Concept of a compatibility matrix (Table 4.11, Page 98)
Backward compatibility and forward compatibility
6/23/2018
Black Box Testing

43. User Documentation Testing
To check if what is stated in the document is available in the product
To check if what is there in the product is available in the document
Documents include user guides, installation guides, set-up guides, read-me files, software release notes and on-line help
6/23/2018
Black Box Testing

44. 6/23/2018
Black Box Testing

45. Benefits of Documentation Testing
User documentation testing aids in highlighting problems that have been overlooked during reviews.
High-quality user documentation ensures the consistency of documentation and product, thus minimizing possible defects reported by customers.
Results in less difficult support calls.
Contributes to better customer satisfaction and better morale of support staff.
New programmers and testers who join a project group can use the documentation to learn the external functionality of the product.
Customers need less training and can proceed more quickly to advanced training and product usage.
Reduced training costs for customers.
6/23/2018
Black Box Testing

46. Domain Testing
6/23/2018
Black Box Testing

47. Domain Testing
Have “domain people” perform tests by using the software
Business flow determines the test, not “logic” or “steps”
Tests what the users do on a typical business day
Captures the typical problems faced by users (not necessarily captured in SRS)
6/23/2018
Black Box Testing

48. When to Use What… 6/23/2018 Black Box Testing
When you want to test scenarios that have…
The most effective black box testing technique is likely to be…
Output values are dictated by certain conditions depending upon values of input variables
Decision Tables
Input values being in ranges, with each range exhibiting a particular functionality
Boundary Value Analysis
Input values being divided into classes (like ranges, list of values, etc), with each class exhibiting a particular functionality
Equivalence Partitioning
Positive and negative testing
Workflows, process flows or language processors
Graph based testing
To ensure that requirements are tested and met properly
Requirements Based Testing
To test using the domain expertise rather than the product specification
Domain Testing
To ensure that the documentation is consistent with the product
Documentation Testing
6/23/2018
Black Box Testing

49. Black Box Testing: Objectives (Recap)
To understand:
What black box testing is
The differences between black box testing and white box testing
Different techniques of black box testing
6/23/2018
Black Box Testing