CS 217 Software Verification and Validation Week 6, Summer 2014 Instructor: Dong Si

REVIEW OF MIDTERM

Statistic of Midterm n Highest Score = 78 n 65 – 78 = A n 50 – 64 = B 3

Testing Principles (1) – p54 of week1 n Testing can demonstrate only the presence of defects and not their absence –Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness. n Exhaustive testing is impossible –Exhaustive testing (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts.

Testing Principles (3) – p56 of week1 n Testing is context dependent –Testing is done differently in different contexts. For example, military software is tested differently from an business site.

Motivation – p21 of week2 n Logic became popular in the early 20 th century among philosophers and mathematicians n What constitutes a correct proof in Mathematics? n Some ‘correct’ proofs were later disproved by other mathematicians n Concept of logic helps us to figure out what constitutes a correct argument and what constitutes a wrong argument 6

p28 of week2  Propositions - a statement that is either true or false.  For every proposition p, either p is T or p is F  For every proposition p, it is not the case that p is both T and F 7

Validation & Verification – p14 of week1 n Validation : Have we built the right software? n i.e., do the requirements satisfy the customer? n (This is dynamic process for checking and testing the real product. Software validation always involves with executing the code) n Verification : Have we built the software right? n i.e., does it implement the requirements? n This is static method for verifying design, code. Software verification is human based checking of documents and files Introduction to Software Testing, Edition 2 (Ch 1) © Ammann & Offutt 8

Introduction to Software Testing, Edition 2 (Ch 1) © Ammann & Offutt 9 n Software Fault : A static defect in the software n Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior n Software Error : An incorrect internal state that is the manifestation/expression of some fault Faults in software are equivalent to design mistakes in hardware. Software does not degrade. Software Faults, Errors & Failures – p21 of week1

Unit Test (Component Level Test) – p65 of week1 n Individual components are tested independently to ensure their quality. The focus is to uncover errors in design and implementation, including 10 The focus is to uncover errors in: - data structure in a component - program logic and program structure in a component - component interface - functions and operations of a component Operations and Functions with I/O White-box interface input output Internal logic, data, structure output input interface operation Black-box

Integration Testing – p66 of week1 n A group of dependent components are tested together to ensure their the quality of their integration unit Design and construction of software architecture - Integrated functions or operations at sub-system level - Interfaces and interactions between them - Resource integration and/or environment integration Component #1 Operations and Functions with I/O input interface operation Component #2 Operations and Functions with I/O output interface operation

System Testing - p67 of week1 n The system software is tested as a whole. It verifies all components mesh properly to make sure that all system functions and performance are achieved in the target environment. 12 The focus areas are: - System functions and performance - System reliability and recoverability (recovery test) - System installation (installation test) - System behavior in the special conditions (stress and load test) - System user operations (acceptance test/alpha test) - Hardware and software integration and collaboration - Integration of external software and the system When a system is to be marketed as a software product, a testing process called beta testing is often used. System (Operations & Functions & Behavior) User interface UserExternal interfaces

Rules of Natural Deduction – p45 of week2 n FR 3 (De Morgan’s law) ~(p v q) = (~p) ^ (~q) ~(p ^ q) = (~p) v (~q) n FR 4 (Law of contrapositive) p q = (~q ~p) n FR 5 (Double Negation) ~(~p) = p 13

Examples of Arguments – p46 of week2 n If a baby is hungry, then the baby cries. If the baby is not mad, then he does not cry. If a baby is mad, then he has a red face. Therefore, if a baby is hungry, then he has a red face. n Model this problem!! n h: a baby is hungry c: a baby cries m: a baby is mad r: a baby has a red face 14 h c ~m ~c m r... h r h c c m m r... h r

Bonus Question – p39 of week3 n What happened if switch AND with OR logic in the program: fun (x, y) { z = 1; IF ((x>z) OR (y>z)) THEN z = 0; Output z; } 15 1.Fun (0, 0) 2.Fun (2, 0) 3.Fun (0, 2) 4.Fun (2, 2) 5.Fun (8, 9) Function coverage: Statement coverage: Branch coverage: Condition coverage:

Example 1: compare two numbers – p50 of week3 n Function ‘compare (x, y)’ n Inputs: Two numbers – x and y n Outputs: A larger number between x and y 16 z = Compare (x, y) (x, y) z

– p51 of week3 17 Equivalence Classes: { (x, y) | x < y } { (x, y) | x > y } { (x, y) | x = y } { input other than a pair of numbers, “as&%dfget^$(&w” } Valid inputs Invalid inputs

18 Valid (x, y) Input Space x = y x < y x > y Three test cases: (1, 2) (8, 8) (100, 30) Plus one test cases: (^&%*) --- ERROR – p52 of week3

P53 of week3 - Example 2: Loan application 19 Customer Name Account number Loan amount requested Term of loan Monthly repayment Term: Repayment: Interest rate: Total paid back: 6 digits, 1st non-zero $500 to $ to 30 years Minimum $ chars. Choosing (or defining) partitions seems easy, but is easy to get wrong…

20 Customer name Number of characters: invalidvalidinvalid 1 Valid characters: Any other A-Z a-z -’ space

21 Loan amount invalidvalidinvalid 499

WHITE-BOX TESTING

Styles of Testing n Testing traditionally can be conducted in two styles n Black-Box testing Try to choose "smart" tests based on the requirements, without looking at the code. n White-Box testing Try to choose "smart" tests based on the structure of the code, with minimal reference to the requirements. 23

Definition of White-Box Testing n Testing based on analysis of internal logic (design, code, etc.). (But expected results still come from requirements.) n Use different techniques to check every visible path of the source code to minimize errors. n It is the ability to know which line of the code is being executed and being able to identify what the correct output should be. 24

Characters of WB testing n Exploits the structure, control or data flow of programs in order to design test cases. n Typically performed during coding. n Allows to test parts of the program, since you know the structure. (with black ‐ box is not possible) n Allows to cover systematically every part of the program. 25

WB Testing Techniques n Logic coverage: (learned in previous class)  Statement coverage  Branch coverage  Condition coverage  … n Dataflow based testing / Path coverage: all program paths have been traversed at least once 26

Pseudo code & Control/Dataflow Graphs 27 “nodes” “edges” Input output Absolute (x) { IF (x>=0) THEN y = x; ELSE IF (x<0) THEN y = -x; Output y; } x IF (x>=0)ELSE IF (x<0) y = x; y = -x; y

Example 28 Fun (x, y) { z = 1; IF ((x>z) OR (y>z)) THEN z = 0; Output z; } Test cases: 1.Fun (0, 0) 2.Fun (2, 0) 3.Fun (0, 2) 4.Fun (2, 2) 5.Fun (8, 9)

Control/Dataflow of the example 29 Fun (x, y) { z = 1; IF ((x>z) OR (y>z)) THEN z = 0; Output z; } Input x, y z = 1; IF ((x>z) OR (y>z)) z = 0; ELSE z = 1; Output z; Test cases: 1.Fun (0, 0) 2.Fun (2, 0) 3.Fun (0, 2) 4.Fun (2, 2) 5.Fun (8, 9)

Statement coverage n Has each statement in the program been executed? 30 √ √ √

Branch coverage n Has each branch of each control structure been executed? n For example, given an if statement, have both the T and F branches been executed? n Another way of saying this is, has every edge in the program been executed? 31

Condition coverage n Has each Boolean sub-expression evaluated both to true (T) and false (F) ? 32

Condition coverage Fun (x, y) { z = 1; IF ((x>z) OR (y>z)) THEN z = 0; Output z; } Test cases: 1.Fun (0, 0) 2.Fun (2, 0) 3.Fun (0, 2) 4.Fun (2, 2) 5.Fun (8, 9) X>1 Y>1 T, F ?

Exercise n Consider a program ‘Compare’ which compare the value between two numbers. The inputs are two real numbers x and y, the program outputs the larger number between x and y. 34 Compare (x, y) { IF (x>y) THEN z = x; ELSE IF (x==y) THEN z = x; ELSE IF (x<y) THEN z = y; Output z; }

Exercise - Control/Data Flow Graphs n I have provided you the Pseudo code, n Can you draw a Control/Dataflow Graphs? n Try it! This is a bonus question with +5 points towards to FINAL!! 35

Exercise - Statement Coverage n Statement Coverage requires that each statement will have been executed at least once. What is the minimum number of test cases required to achieve statement coverage for the program Compare (x, y) ? 36

Exercise - Branch Coverage n Branch Coverage requires that each branch will have been traversed at least once. What is the minimum number of test cases required to achieve branch coverage for the program Compare (x, y) ? 37

Condition Coverage n Condition Coverage requires that each condition (sub-expression) will have been True at least once and False at least once. n What is the relationship between Branch and Condition Coverage?

Condition / branch coverage IF ( AND ) THEN … ELSE … 39 X>0 Y>0 T F T, F ?

Condition Coverage example – Combination ! IF A or B THEN s1; ELSE s2; ABBranch test 1TFtrue test 2FFfalse test 3TTtrue test 4FTtrue Combinations of condition values: TT, TF, FT, FF

Exercise IF (A or B or C) THEN s1; ELSE s2; What is the minimum number of test cases required to achieve condition coverage for the above pseudo code?

Dataflow based testing -> Path coverage n All program paths have been traversed at least once. 42 Input x, y z = 1; IF ((x>z) OR (y>z)) z = 0; ELSE z = 1; Output z; Two paths !

Path coverage - Example 1 43 if a then s1 else if b then s2 else if c then s3 else s4 end_if_then_else # Paths: ___ # Branches: ___

WB Testing Techniques n Logic coverage: (learned in previous class)  Statement coverage  Branch coverage  Condition coverage  … n Dataflow based testing / Path coverage: all program paths have been traversed at least once 44

Coming Up Next week… Lab 2: White-box Testing n Preparation:  Go over the slides of WB testing  Understand “Code coverage” & “Data flow / Path”