PROGRAMMING TESTING B MODULE 2: SOFTWARE SYSTEMS 22 NOVEMBER 2013

Week 1 Values, conditions Classes, objects Week 2 Specifications Testing Week 3 Abstraction, inheritance Week 4 Interfaces, abstract classes Arrays Week 5 Collections Generic structures Week 6 Exceptions Stream I/O GUIs Week 7 Concurrency Networking Week 8 Security Week 9/10 Project Software Systems - Programming 2 OVERVIEW PROGRAMMING LINE

 System testing & unit testing  Test plan  Test framework  Test coverage Nino & Hosch: Chapter 6 Tool support: Emma (EclEmma for Eclipse) Self study: Nino & Hosch: Chapters 7 & 8 Software Systems - Programming 3 CONTENTS

 System testing (functional testing)  Complete system, final configuration  Test system behavior, e.g., use cases  Typically performed by testing team  Unit testing  Class, method, etc. in isolation  Test unit’s behavior  Typically performed by developer of unit  Further tests: Performance ∼, Stress ∼, Platform ∼, Usability ∼  Out of scope for this course Software Systems - Programming 4 OVERVIEW: TESTING TERMINOLOGY

 Test whether the system or unit  Behaves as intended  When used in the specified way  Plan:  List of execution scenarios  Describe how to perform scenario  Describe expected outcome Software Systems - Programming 5 TEST PLAN

 Use cases as execution scenarios, consider  Alternative flows in use cases  Different values used in use cases  Additional requirements  Treat system as “black box”  Internal structure is unknown  Validation by observing behavior Software Systems - Programming 6 TEST PLAN: SYSTEM TESTING

 One test plan per class or method, can also be  Per group of logically related methods  Per group of classes in the same hierarchy  Consider method contracts, test that  The postcondition of methods holds  When the precondition is satisfied  Treat method as “white box”  Internal structure is known  E.g., consider conditionals  Validation: by inspecting method result and/or instance/class variables Software Systems - Programming 7 TEST PLAN: UNIT TESTING Lecture will further focus on unit testing. For the project, you need to perform both, system and unit tests! Lecture will further focus on unit testing. For the project, you need to perform both, system and unit tests!

 Identify relevant cases Example: Lock  Four cases  Attempt to open in open state  Attempt to open in closed state  Attempt to close in open state  Attempt to close in closed state  Actually many more situations: Do we have to test for all possible values?  1000 possibilities for lockCode  1000 possibilities for code passed to enterDigits →isOpen() →isOpen() code correct →not isOpen() Software Systems - Programming 8 TEST CASES Lock -int lockCode +boolean isOpen() +void close() +void enterDigits(int code) > 2,000,000 combinations

 Do we have to test for all possible values?  NO! Pick representative values  What is representative?  Group values for which the unit under test behaves the same  Pick samples from this group  Random value  Boundaries (e.g,. maximum and minimum)  E.g,. for the Lock: pick 0, 123, 999 Software Systems - Programming 9 TEST CASES

 Developers are like to make mistakes at boundaries  So called off-by-one errors Example: ModuloCounter  Specification  Count from 0 up to 9  When increasing with a value of 9, reset to 0 instead  Test  Invoke increase: zero times, five times, nine times, ten times Software Systems - Programming 10 BOUNDARY CASES ModuloCounter -int value +int getValue() +void increase() this.value = this.value + 1; if (value >= 9) { this.value = 0: } off-by-one error

 Primarily test for correct behaviour when input is within specified bounds  Sometimes we want to test that nothing bad happens when input is wrong  E.g., data entered through the user interface  Test robustness of software  Can only be tested if the system behavior for wrong input is specified  E.g., usage of default values instead of wrong input  Beware: documenting behaviour outside specified precondition encourages to ignore preconditions!  Specification of exceptional behaviour: see week 6 Software Systems - Programming 11 TESTING ILLEGAL DATA

 One class per test plan  Implement each test case as method  Test method uses unit under test and asserts correct behaviour  Compare actual outcome with expected outcome  Often test cases share preparatory tasks, e.g., creating instance of class under test  Extract shared code into setUp() method  setUp() method executed before each test method  Establishes defined initial situation Software Systems - Programming 12 COMMON STRUCTURE OF UNIT TEST

 JUnit (  Provides framework for writing and running unit tests  Uses Java features not introduced in this course  Provided: simplified version of Junit  Main class ( Test )  Test class per class-under-test  Main class is executable (has main method)  Instantiates and test classes and invokes their method runTest()  Test classes implement test plan  Method runTest() must:  invoke each test method  invoke setUp() before each test method Software Systems - Programming 13 TEST FRAMEWORK

 If my code passes all tests, I’m done!  Is this correct?  How do I assure a good quality of my tests?  Systematically develop tests based on contracts  Review tests  Measure coverage  Other techniques, not covered in this course Software Systems - Programming 14 QUALITY OF TESTS

 Statement  Each statement executed at least once  Branch  E ach branch traversed (and every entry point taken) at least once  Path  All program paths traversed at least once Software Systems - Programming 15 TYPES OF COVERAGE

 100% coverage difficult to achieve  Especially for branch or path coverage, too many test cases would be needed  Synthetic code generated by compiler  Dead code (should be eliminated when discovered)  Code handling exceptional situations  High statement coverage generally possible (e.g., > 80%) Software Systems - Programming 16 GOALS FOR CODE COVERAGE

 Open-source tool  Supports class, method, and line coverage.  “Fractional” line coverage supported, but not branch coverage.  Eclipse plugin EclEmma also available  Software Systems - Programming 17 EMMA

Software Systems - Programming 18 EMMA COVERAGE REPORT

Software Systems - Programming 19 EMMA SOURCE CODE ANNOTATIONS

Software Systems - Programming 20 FRACTIONAL LINE COVERAGE Only part of conditional executed Loop increment not executed

 Java compiler may emit multiple copies of code, e.g., for instance field initializer  Emma expects coverage of all copies Software Systems - Programming 21 ANOMALIES IN REPORTED COVERAGE  Java compiler may generate default code, e.g., for missing constructors  Emma also reports coverage of such code

 Iterative process  Perform unit tests always after functionality is complete  Test from the beginning of the implementation  Extending implementation requires extending test plan  Test results lead to further coding  Test coverage results lead to extending test plan  Philosophies  “Code a little, test a little”  “Test a little, code a little” (test-driven development, test first) Software Systems - Programming 22 TESTING / DEVELOPMENT PROCESS

 Test plan  System tests based on use cases  Unit tests based on method contracts  Test representatives and boundary cases  Confirm quality of your tests by measuring their code coverage Software Systems - Programming 23 MAIN POINTS