(1) Verification Methods CS2110: SW Development Methods Reading: Study Section in the MSD text These slides supplement the textbook – These touch on other important methods, but focus on execution-based testing

(2) Some Definitions Verification: showing that something meets specifications –We verify “against” something –Important: if you don’t have requirements specifications documented, then you can’t verify! –If behavior is not defined, then what counts as incorrect behavior? Verification can happen on any kind of software work- product –designs, code, etc.

(3) Verification Techniques At least four approaches to verifying code quality: 1.Execution-based Testing (running tests) 2.Inspections –People examine the work product by eye 3.Static Analysis by a tool –For C programs, an old tool called “lint” –For Java programs, some nice tools integrate with Eclipse: CheckStyle and PMD 4.Formal Correctness Proofs –Use mathematics to demonstrate the desired property

(4) Execution-based Testing Some Basic Definitions: Fault: What we think of as a bug –Something in the code that’s wrong Failure: Observed incorrect behavior of a system resulting from a fault Error: Mistake made by a developer A possible definition of execution-based testing: –Process of inferring certain behavioral properties of a product based, in part, on results of executing it in a known environment with selected inputs.

(5) Famous and Important Quote Program testing can effectively show the presence of bugs but is hopeless for showing their absence. –Edsger Dijkstra –Famous computer scientist –Turing Award winner –Wrote "Go To Statement Considered Harmful“ (1968) –Shortest-path algorithm

(6) Types of Testing Unit testing –An individual class or function is tested thoroughly –Done as part of coding process –Done by the programmer him/herself –Usually by a test harness (or driver) or using a test-class Integration testing –Multiple parts of a system are combined and tested to see if they work together Acceptance testing –The customer is involved. (Alpha or beta testing)

(7) Approaches to Unit Testing Two approaches –Test only based on inputs, outputs (i.e. the specification) without looking at the code –Test based on the logic of the code (must study it) The 1 st of these is called black-box testing The 2 nd of these is glass-box or white-box testing Some principles of glass-box testing –Make sure every line (or “case”) gets executed –Make sure loops get done zero times, >zero times –Etc.

(8) Limitations of Glass-Box Testing Why not fully exercise our code? Logic of a piece of code defines a set of possible execution paths, “flow-paths,” through the function –Think of a flow-chart –The inputs control which path is taken A good set of test data might make sure every possible path is followed –This tests every possible behavior Problem: for even small programs with loops and conditionals, the total number of paths becomes too large too quickly –See MSD text, p

(9) Black-Box Testing This means testing based on inputs to the code- module without basing tests on code’s internals –You can’t see inside a black-box –Note: MSD’s definition on p. 26 isn’t the usual one Choose test cases for common problems –Boundary values, boundary conditions –Off-by-one errors –Incorrect inputs Important: You have to know in advance what the expected result of any test should be!

(10) Lab on Monday/Tuesday The JUnit (v. 3) framework is a tool to help people generate and format unit tests in a systematic way. Stand-alone and also built into Eclipse Example JUnit test –Class contains all tests –setUp() performs preconditions –“test…” is an individual test case –“assert” checks a program condition public class CoffeeMakerTest extends TestCase { private CoffeeMaker cm; private Recipe recipe1; public void setUp() { cm = new CoffeeMaker(); recipe1 = new Recipe(); recipe1.setName("Coffee"); recipe1.setPrice(50); recipe1.setAmtCoffee(6); recipe1.setAmtMilk(1); } public void testAddRecipe() { assertTrue(cm.addRecipe(recipe1)); }

(11)

(12) When we test a class’ method to see if its loop goes to size-1 and not size, is this clearly an example of: 1.A black box test 2.A glass-box test 3.A unit test 4.Both 1 and 3 5.Both 2 and 3

(13) Planning Your Tests How do we plan a “test case”? Define three parts first: –A purpose (why are we doing this particular test?) –An input –An expected result We execute the test to see if observed output matched the expected result –It’s really just like a scientific experiment When testing, have a plan (at least in your head) –Don’t test blindly or randomly

(14) Planning Your Tests Poorly specified test cases… Test IDDescriptionExpected Results Actual Results 1Player 1 rolls dice and moves. Player 1 moves on the board. 2Player 2 rolls dice and moves. Player 2 moves on the board.

(15) Planning Your Tests Why are these test cases better than the previous ones? Test IDDescriptionExpected Results Actual Results 1Precondition: Game is in test mode, GameBoard is loaded, and game begins. Number of Players: 2 Player 1 dice roll: 3 Player 1 moves to square Blue 3. 2Precondition: TestID 1 completed successfully. Player 2 dice roll: 3 Player 2 moves to square Blue 3.

(16) Unit Testing: Boundary values Often programs misbehave because we make mistakes dealing with the “extremes” Examples: –The last item or first item in a collection –The start or end of a range of values Equivalent classes of test cases –Don’t have test cases or data that test the same situation (e.g. the middle of a range) Boundary or extreme conditions –An empty list, an empty file –One item in a list or file

(17) Unit Testing: Off-by-one errors Related to boundary conditions Using the nth value instead of n-1 Using 0 instead of 1 Executing a loop body once when it the body should not be executed at all –Wait – is this one a black-box testing method? Why or why not? Etc.

(18) Unit Testing: Invalid Inputs Rely on pre-conditions defined for your functions –They specify what the agreement is about what you are responsible for when its time to handle mis-use If a function is mis-used in this way –Return an error condition if the interface provides a way to do this –Otherwise, halt program! Better than producing unexpected results Use assertions (more later!)

(19) Example: Program with Student and Course classes Homework 1 this term –Define class Student methods include: –Define class Course methods include: Focus on defining: –Black-box test-cases for the method boolean drop(Course c) in class Student Let’s make a list of possible test-cases for this method!

(20) Example of Full Test-Case Description Test Case #1: Drop last course in list with >1 item –Purpose: Drop Course c for Student s where that course is the last course stored in the Student object's courses list. This tests the boundary condition of last item in list for Student.drop(c). –Input: Student s is enrolled in three courses, and the course to be dropped is the last of these stored in the list. –Expected output: The Student s is unchanged except that only two courses are stored for that student. Also, the list of students stored for that course is altered by having that student removed.

(21) Other Test Cases for this Program There are a lot! Boundary conditions? –for Student.drop(): end of list, start of list –same for Course.drop() –Question: affected by whether you used ArrayList instead of Java arrays? –Note: also test “typical” case, item in middle of list –Empty lists: Course.cancel(), Student.dropAll() Off-by-one errors? –Really included in boundary condition testing here

(22) Other Test Cases for this Program Invalid inputs? –See the specifications! –Test return values For boolean, test both success and failure Remember: make it fail! Test of equals() methods? –If you get this wrong, trouble! –Easy to test. (Easy to forget to.)

(23) Good or Bad Test? 1.Good Test? 2.Poor Test? Student s adds a course and the course appears in the schedule

(24) Good or Bad Test? 1.Good Test? 2.Poor Test? Student s currently is in the system and currently has no courses. Student s adds a course number 394 and it appears at the end of the schedule.

(25) Good or Bad Test? 1.Good Test? 2.Poor Test? Student s has 4 courses, drops 3, adds 1. Check if the student is then deleted from the system.

(26) Extra Slides

(27) #2: Inspections: We Might See These Later Many different “styles” or approaches –Walk-throughs, Inspections Advantages: –You can inspect all of a product but usually it is impossible to test all of a product. –Applicable to all work products (e.g. code, designs, requirements documents, etc.) –Fairly easy: training, application, follow-up Disadvantages: –Staff time is expensive –Works best if follows a rigorous process