Software Testing For CSE 3902 Matt Boggus

Terms: Setup / Exercise / Verify / Teardown Setup - whatever needs to be prepared before the code can be run Exercise - run the code we want to test Verify - compare the result of the run with some expected condition Teardown - cleanup all the extra stuff we used for testing so that the system is in the same state as it was before we started the current test (the state from before the Setup step).

Terms: code coverage Function coverage - Has each function been called? Statement coverage - Has each statement been executed? Branch coverage - Has each branch (i.e. if-else blocks or cases of a switch) been executed? Path coverage - Has every possible route through a given part of the code been executed? Loop coverage - Has every possible loop been executed zero times, once, and more than once? State coverage - Has each state in a finite-state machine been reached and explored?

Test Driven Development Write tests first Write minimal amount of code to pass the test Refactor code to fit quality standards

Mutation testing Given an existing implementation and set of unit tests Modify/Mutate one statement in existing code, ex: Arithmetic operators: + becomes a – Boolean operators: < becomes a <= Values: numeric literal or constant is increased or decreased by 1 Adding or removing statements Generate X mutant code samples and run unit tests If one of the tests fails the mutant has been “killed” Killed mutants / total mutants measures usefulness of current unit tests

Testing Anti-patterns Having dependencies between test cases Order that tests are executed shouldn’t matter Failing on one test should not affect the pass/fail of the next test Testing for run-time speed/performance Slow running tests Over specification – starting conditions are too constrained Under specification – starting conditions are not constrained enough The ice cream cone and cupcake – too much reliance on manual testingice cream conecupcake