White Box and Black Box Testing Tor Stålhane

What is White Box testing White box testing is testing where we use the info available from the code of the component to generate tests. This info is usually used to achieve coverage in one way or another – e.g. Code coverage Path coverage Decision coverage Debugging will always be white-box testing

Coverage report. Example – 1

Coverage report. Example – 2

McCabe’s cyclomatic complexity Mathematically, the cyclomatic complexity of a structured program is defined with reference to a directed graph containing the basic blocks of the program, with an edge between two basic blocks if control may pass from the first to the second (the control flow graph of the program). The complexity is then defined as:structured programdirected graphbasic blocks v(G) = E − N + 2P v(G) = cyclomatic complexity E = the number of edges of the graph N = the number of nodes of the graph P = the number of connected componentsconnected components

Graph example We have eight nodes – N = 8 – nine edges – E = 9 – and we have only one component – P = 1. Thus, we have v(G) = 9 – = 3.

Simple case - 1 S1; IF P1 THEN S2 ELSE S3 S4; One predicate – P1. v(G) = 2 Two test cases can cover all code S4 S3S2 S1 P1

Simple case – 2 S1; IF P1 THEN X := a/c ELSE S3; S4; One predicate – P1. v(G) = 2 Two test cases will cover all paths but not all cases. What about the case c = 0? S4 S3 S1 a/c P1

Statement coverage – 1 IF in_data > 10 {out_data = 4;} ELSE {out_data = 5;} IF out_data == 8 {update_panel();} How can we obtain full statement coverage? P1 P2 S2 S1 S3 empty

Statement coverage – 2 out_data = 0 IF in_data > 10 {out_data = 4;} update_panel(); If we set in_data to 12 we will have full statement coverage. What is the problem?

Decision coverage IF (in_data > 10 OR sub_mode ==3) {out_data = 4;} ELSE {…..} P1 is really two decisions P1-1: in_data > 10 P1-2: sub_mode == 3 We need to cover both decisions P1 P1-1 P1-2 S1 empty

Using v(G) The minimum number of paths through the code is v(G). As long as the code graph is a DAG – Directed Acyclic Graph – the maximum number of paths is 2**|{predicates}| Thus, we have that V(G) < number of paths < 2**|{predicates}|

Problem – the loop S4 S2 S1 P1 S5 S3 P2 S1; DO IF P1 THEN S2 ELSE S3; S4 OD UNTIL P2 S5; No DAG. v(G) = 3 and Max is 4 but there is an “infinite” number of paths.

Nested decisions P1 P2 S5 S4 S6 S3 S2 S1 S1; IF P1 THEN S2 ELSE S3; IF P2 THEN S4 ELSE S5 FI S6; v(G) = 3, while Max = 4. Three test case will cover all paths.

Using a decision table – 1 A decision table is a general technique used to achieve full path coverage. It will, however, in many cases, lead to over-testing. The idea is simple. 1.Make a table of all predicates. 2.Insert all combinations of True / False – 1 / 0 – for each predicate 3.Construct a test for each combination.

Using a decision table – 2 P1P2P3Test description or reference

Using a decision table – 3 Three things to remember: The approach as it is presented here is only practical for Situations where we have binary decisions. Small chunks of code – e.g. class methods and small components. It will be too laborious for large chunks of code. Note that code that is difficult to reach – difficult to construct the necessary predicates – may not be needed as part of the system.

Decision table example – binary P1 P2 S5 S4 S6 S3 S2 S1 P1P2Test description or reference 00 S1, S3, S5, S6 01 S1, S3, S4, S6 10 S1, S2, S6 11 The last test is not necessary

Decision table example – tertiary S2 S1 S4 S3 S6 S10 S5 S9 S8S7 P3P2 P1 P2P3 Test description or reference 000S1, S3, S9 001S1, S4, S9 002S1, S5, S9 010S1, S3, S9 011S1, S4, S9 012S1, S5, S9 100S2, S6, S S2, S7, S

What about loops Loops are the great problem in white box testing. It is common practice to test the system going through each loop 0 times – loop code never executed 1 time – loop code executed once 5 times – loop code executed several times 20 times – loop code executed “many” times

Loop diagram for (i; P1; i++) {S1} The empty branch is taken when P1 is False > S1 P1

Error messages Since we have access to the code we should 1.Identify all error conditions 2.Provoke each identified error condition 3.Check if the error is treated in a satisfactory manner – e.g. that the error message is clear, to the point and helpful for the intended users.

What is Black Box testing Black box testing is also called functional testing. The main ideas are simple: 1.Define initial component state, input and expected output for the test. 2.Set the component in the required state. 3.Give the defined input 4.Observe the output and compare to the expected output.

Info for Black Box testing That we do not have access to the code does not mean that one test is just as good as the other one. We should consider the following info: Algorithm understanding Parts of the solutions that are difficult to implement Special – often seldom occurring – cases.

Clues from the algorithm We should consider two pieces of info: Difficult parts of the algorithm used Borders between different types of solution – e.g. if P1 then use S1 else use S2. Here we need to consider if the predicate is – Correct, i.e. contain the right variables – Complete, i.e. contains all necessary conditions

Black Box vs. White Box testing We can contrast the two methods as follows: White Box testing – Understanding the implemented code. – Checking the implementation – Debugging Black Box testing – Understanding the algorithm used. – Checking the solution – functional testing

Testing real time systems W-T. Tsai et al. have suggested a pattern based way of testing real time / embedded systems. They have introduced eight patterns. Using these they have shown through experiments that, using these eight patterns, they identified on the average 95% of all defects. We will have a look at three of the patterns. Together, these three patterns discovered 60% of all defects found

Patterns and coverage (from Tsai)

Basic scenario pattern - BSP Check for precondition Check post-condition PreCondition == true / {Set activation time } IsTimeout == true / [report fail] PostCondition == true / [report success]

BSP – example Requirement to be tested: If the alarm is disarmed using the remote controller, then the driver and passenger doors are unlocked. Precondition: the alarm is disarmed using the remote controller Post-condition: the driver and passenger doors are unlocked

BSP test 1.Generate input to make precondition a.False => nothing happens b.True => input activation time 2.Check system response a.Timeout = true => fails b.Timeout = false and post condition OK => OK c.Timeout false and post condition not OK => fails

Key-event service pattern - KSP Check for key event Check post-condition Check precondition PreCondition == true PostCondition == true / [report success] KeyEventOccurred / [SetActivationTime] IsTimeout == true / [report fail]

KSP- example Requirement to be tested: When either of the doors are opened, if the ignition is turned on by car key, then the alarm horn beeps three times Precondition: either of the doors are opened Key-event: the ignition is turned on by car key Post-condition: the alarm horn beeps three times

KSP test 1.Generate key event a.Not right event => wait b.Right event => set activation time 2.Check precondition a.Not OK => wait b.OK => check post condition 3.Check post condition a.Timeout = true => fails b.Timeout = false and post condition OK => OK c.Timeout false and post condition not OK => fails

Timed key-event service pattern - TKSP Check for key event Check post-condition Check precondition PreCondition == true IsTimeout == true / [report fail] PostCondition == true / [report success] KeyEventOccurred / [SetActivationTime] DurationExpired / [report not exercised]

TKSP – example (1) Requirement to be tested: When driver and passenger doors remain unlocked, if within 0.5 seconds after the lock command is issued by remote controller or car key, then the alarm horn will beep once

TKSP – example (2) Precondition: driver and passenger doors remain unlocked Key-event: lock command is issued by remote controller or car key Duration: 0.5 seconds Post-condition: the alarm horn will beep once

TKSP test 1.Generate key event a.Not right event => wait for new event or duration expired b.Right event => set activation time 2.Check precondition a.Not OK => wait b.OK => check post condition 3.Check post condition a.Timeout = true => fails b.Timeout = false and post condition OK => OK c.Timeout false and post condition not OK => fails

Test automation – 1

Test automation – 2 1.Generate stimuli 2.Get necessary data 3.Collect events 4.Check events 1 234

Test automation example – BSP 1.Illegal command a.Generate something b.Nothing happens => test OK Post condition = “doors unlocked” => test fails 2.Legal command a.Generate “alarm disabled” b.Activation time = T c.Timeout = True => test fails Timeout = false post condition = “doors unlocked” => test OK

Needs, models and methods – 1 If we say to the developers that they need to do e.g. unit testing this is just a statement of need, it is not a statement of how – the method to use. If we say that they should use white box testing, this helps a little, but there is still a lot of freedom when it comes to how.

Example – white box testing White box testing => Static white box testing => – Code inspection – Code walkthrough Dynamic white box testing => – Statement coverage – Path coverage – Decision coverage – All define – use coverage

Needs, models and methods – 2 We can use the following structure to organize the decisions: Strategy or need => Technique => Method This will be applicable on all levels, e.g. acceptance test, system test, integration tests and unit tests. For unit test we can for instance use: Unit test => White box test => Path coverage