CS 240, Prof. Sarwar Slide 1 CS 240: Software Project Fall 2003 Sections 1 & 2 Dr. Badrul M. Sarwar San Jose State University Lecture #18

CS 240, Prof. Sarwar Slide 2 Agenda  Software Testing Principles  Integration Testing  Usability Testing  Practical Testing aspects—Fault-based Testing Approach  System Testing

CS 240, Prof. Sarwar Slide 3 Integration testing  Tests complete systems or subsystems composed of integrated components  Integration testing should be black-box testing with tests derived from the specification  Main difficulty is localising errors  Incremental integration testing reduces this problem

CS 240, Prof. Sarwar Slide 4 Incremental integration testing

CS 240, Prof. Sarwar Slide 5 Product with 13 Modules

CS 240, Prof. Sarwar Slide 6 Approaches to integration testing  Big bang testing  Top-down testing  Start with high-level system and integrate from the top-down replacing individual components by stubs where appropriate  Bottom-up testing  Integrate individual components in levels until the complete system is created  In practice, most integration involves a combination of these strategies known as  Sandwich testing

CS 240, Prof. Sarwar Slide 7 Big Bang Integration Testing  It assumes that all components are first tested individually and then tested together as a single system  No additional test stubs or drivers needed  It sounds simple, but it’s expensive  if a test uncovers a failure (which we expect to happen), it is difficult to pinpoint the specific component responsible for the failure  In addition, it is impossible to distinguish failures in the interface from failures within a component  The better approach  Incremental or phased (or snowballing) integration

CS 240, Prof. Sarwar Slide 8 Test Stubs and Test Drivers  To test module a, modules b, c, d must be stubs  Empty module, or  Prints message ("Procedure radarCalc called"), or  Returns precooked values from preplanned test cases  To test module h on its own requires a driver, which calls it  Once, or  Several times, or  Many times, each time checking the value returned  Testing module d requires a driver and two stubs

CS 240, Prof. Sarwar Slide 9 Problems with: Implementation, Then Integration Approach  Problem 1  Stubs and drivers must be written, then thrown away after module testing is complete  Problem 2  Lack of fault isolation  A fault could lie in any of 13 modules or 13 interfaces  In a large product with, say, 103 modules and 108 interfaces, there are 211 places where a fault might lie  Solution to both problems  Combine module and integration testing  “Implementation and integration phase”

CS 240, Prof. Sarwar Slide 10 Top-down Implementation and Integration  If module m1 calls module m2, then m1 is implemented and integrated before m2  One possible top-down ordering is a, b, c, d, e, f, g, h, i, j, k, l, m  Another possible top-down ordering is a [a]b, e, h [a]c, d, f, i [a, d]g, j, k, l, m

CS 240, Prof. Sarwar Slide 11 Top-down Implementation and Integration: Advantages  Fault isolation  A previously successful test case fails when mNew is added to what has been tested so far  Stubs not wasted  Each stub is expanded into the corresponding complete module at the appropriate step  Major design flaws show up early  Logic modules include decision-making flow of control  In the example, modules a, b, c, d, g, j  Operational modules perform actual operations of module  In the example, modules e, f, h, i, k, l, m  Logic modules are developed before operational modules

CS 240, Prof. Sarwar Slide 12 Top-down Implementation and Integration: Disadvantages  Problem 1  Reusable modules are not properly tested  Lower level (operational) modules are not tested frequently  The situation is aggravated if the product is well designed  Defensive programming (fault shielding)  Example: if (x >= 0) y = computeSquareRoot (x, errorFlag);  Never tested with x < 0  Reuse implications

CS 240, Prof. Sarwar Slide 13 Bottom-up Implementation and Integration  If module m1 calls module m2, then m2 is implemented and integrated before m1  One possible bottom-up ordering is l, m, h, i, j, k, e, f, g, b, c, d, a  Another possible bottom-up ordering is h, e, b i, f, c, d l, m, j, k, g[d] a[b, c, d]

CS 240, Prof. Sarwar Slide 14 Bottom-up Implementation and Integration: Advantages  Advantage 1  Operational modules are thoroughly tested  Advantage 2  Operational modules are tested with drivers, not by fault shielding, defensively programmed calling modules  Advantage 3  Fault isolation

CS 240, Prof. Sarwar Slide 15 Bottom-up Implementation and Integration: Disadvantages  Difficulty 1  Major design faults are detected late  Solution  Combine top-down and bottom-up strategies making use of their strengths and minimizing their weaknesses

CS 240, Prof. Sarwar Slide 16 Sandwich Implementation and Integration  Logic modules are implemented and integrated top-down  Operational modules are implemented and integrated bottom-up  Finally, the interfaces between the two groups are tested

CS 240, Prof. Sarwar Slide 17 Sandwich Implementation and Integration (contd)  Advantage 1  Major design faults are caught early  Advantage 2  Operational modules are thoroughly tested  They may be reused with confidence  Advantage 3  There is fault isolation at all times

CS 240, Prof. Sarwar Slide 18 Top-down testing

CS 240, Prof. Sarwar Slide 19 Bottom-up testing

CS 240, Prof. Sarwar Slide 20 Testing approaches  Architectural validation  Top-down integration testing is better at discovering errors in the system architecture  System demonstration  Top-down integration testing allows a limited demonstration at an early stage in the development  Test implementation  Often easier with bottom-up integration testing  Test observation  Problems with both approaches. Extra code may be required to observe tests

CS 240, Prof. Sarwar Slide 21  Takes place when modules or sub-systems are integrated to create larger systems  Objectives are to detect faults due to interface errors or invalid assumptions about interfaces  Particularly important for object-oriented development as objects are defined by their interfaces Interface testing

CS 240, Prof. Sarwar Slide 22 Interface testing

CS 240, Prof. Sarwar Slide 23 Interfaces types  Parameter interfaces  Data passed from one procedure to another  Shared memory interfaces  Block of memory is shared between procedures  Procedural interfaces  Sub-system encapsulates a set of procedures to be called by other sub- systems  Message passing interfaces  Sub-systems request services from other sub-systems

CS 240, Prof. Sarwar Slide 24 Interface errors  Interface misuse  A calling component calls another component and makes an error in its use of its interface e.g. parameters in the wrong order  Interface misunderstanding  A calling component embeds assumptions about the behaviour of the called component which are incorrect  Timing errors  The called and the calling component operate at different speeds and out- of-date information is accessed

CS 240, Prof. Sarwar Slide 25 Interface testing guidelines  Design tests so that parameters to a called procedure are at the extreme ends of their ranges  Always test pointer parameters with null pointers  Design tests which cause the component to fail  Use stress testing in message passing systems  In shared memory systems, vary the order in which components are activated

CS 240, Prof. Sarwar Slide 26 Usability Testing (User Interface) Testing  GUIs work as a collection of controls that a human can stimulate via mouse or keyboard  Some controls are simple (e.g., buttons) but some are complex (e.g., list boxes), as they pass control as well as data  Testing data passing controls are challenging  Another concern is the order in which GUI controls are used  As a tester our job would be to understand the events and data combinations that originate and ensure that all interesting cases get tested

CS 240, Prof. Sarwar Slide 27 Attacking GUI  We’ll look into four categories to attack a GUI  input  output  storing of data  performing computation  Strategies in testing GUI input  Apply inputs that force all error messages to occur  Apply inputs that force the software to establish default values  Explore allowable character sets and data types  if you type in a URL named file://c:\AUX in internet explorer v 5.5 the program breaks  Overflow input buffers  We have seen one such boundary value attack on MS Word

CS 240, Prof. Sarwar Slide 28 Attacking GUI  Strategies in testing GUI input  repeat the same input or series of inputs numerous times  Microsoft equation editor fails this test  Force different outputs to be generated for each input  Force invalid outputs to be generated  Win NT (earlier versions) showed 2001 as a leap year  Force properties of an output to change  Ex. MS Powerpoint  Force the screen to refresh  MS Powerpoint fails this test

CS 240, Prof. Sarwar Slide 29 Attacking GUI  Strategies in testing GUI Data and Computation  Apply inputs using a variety of initial conditions  Force a data structure to store too many or too few values  Use alternative ways to modify internal data constraints  Force a function to call itself recursively  Force computation result to be too large or too small  Strategies in testing GUI File system  Fill the file system to its capacity  force the media to be busy or unavailable  Damage the media  Assign an invalid file name  vary or corrupt file contents  change file access permissions etc.

CS 240, Prof. Sarwar Slide 30 System Testing  Unit and Integration testing focus on finding faults in individual components and the interfaces between the components  After integration system testing is performed  There are several system testing activities  Functional testing  Nonfunctional (performance) testing  Pilot testing  Acceptance testing  Installation testing

CS 240, Prof. Sarwar Slide 31 Functional Testing  a.k.a Requirements testing finds differences between the functional requirements and the system.  System testing is a black-box testing  Test cases are derived from the use case model  although usability testing also uses use-case model, there is a subtle difference between them  functional: finds differences between the use case model and the observed system behavior  usability: finds differences between the use case model and the user’s expectation of the system

CS 240, Prof. Sarwar Slide 32 Performance Testing  Finds the differences between the design goals selecte3d during the system design and the observed system  The following tests are performed during performance testing  Stress Testing  volume testing  security testing  timing tests  recovery tests

CS 240, Prof. Sarwar Slide 33 Stress testing  Exercises the system beyond its maximum design load. Stressing the system often causes defects to come to light  Stressing the system test failure behaviour.. Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data  Particularly relevant to distributed systems which can exhibit severe degradation as a network becomes overloaded

CS 240, Prof. Sarwar Slide 34 Pilot Testing  a.k.a Field Test  The system is installed and used by a selected set of users  Useful when the system is built without a specific set of requirements  An alpha test is a pilot test with users exercising the system in the development environment  A beta test is an acceptance test done by a limited number of end users in the target environment

CS 240, Prof. Sarwar Slide 35 Acceptance Testing  There are three ways a client evaluates a system during acceptance testing  Benchmark test  by using a set of test cases under which the system should operate  Competitor test  if the product is replacing an earlier product then it is tested against it  Shadow test  the new and legacy systems run in parallel and their outputs are compared

CS 240, Prof. Sarwar Slide 36  The components to be tested are object classes that are instantiated as objects  Larger grain than individual functions so approaches to white-box testing have to be extended  No obvious ‘top’ to the system for top-down integration and testing Object-oriented testing

CS 240, Prof. Sarwar Slide 37 Testing levels  Testing operations associated with objects  Testing object classes  Testing clusters of cooperating objects  Testing the complete OO system

CS 240, Prof. Sarwar Slide 38 Object class testing  Complete test coverage of a class involves  Testing all operations associated with an object  Setting and interrogating all object attributes  Exercising the object in all possible states  Inheritance makes it more difficult to design object class tests as the information to be tested is not localised

CS 240, Prof. Sarwar Slide 39 Weather station object interface  Test cases are needed for all operations  Use a state model to identify state transitions for testing  Examples of testing sequences  Shutdown  Waiting  Shutdown  Waiting  Calibrating  Testing  Transmitting  Waiting  Waiting  Collecting  Waiting  Summarising  Transmitting  Waiting

CS 240, Prof. Sarwar Slide 40 Object integration  Levels of integration are less distinct in object-oriented systems  Cluster testing is concerned with integrating and testing clusters of cooperating objects  Identify clusters using knowledge of the operation of objects and the system features that are implemented by these clusters

CS 240, Prof. Sarwar Slide 41 Approaches to cluster testing  Use-case or scenario testing  Testing is based on a user interactions with the system  Has the advantage that it tests system features as experienced by users  Thread testing  Tests the systems response to events as processing threads through the system  Object interaction testing  Tests sequences of object interactions that stop when an object operation does not call on services from another object

CS 240, Prof. Sarwar Slide 42 Scenario-based testing  Identify scenarios from use-cases and supplement these with interaction diagrams that show the objects involved in the scenario  Consider the scenario in the weather station system where a report is generated

CS 240, Prof. Sarwar Slide 43 Collect weather data

CS 240, Prof. Sarwar Slide 44 Weather station testing  Thread of methods executed  CommsController:request  WeatherStation:report  WeatherData:summarise  Inputs and outputs  Input of report request with associated acknowledge and a final output of a report  Can be tested by creating raw data and ensuring that it is summarised properly  Use the same raw data to test the WeatherData object

CS 240, Prof. Sarwar Slide 45 Testing workbenches  Testing is an expensive process phase. Testing workbenches provide a range of tools to reduce the time required and total testing costs  Most testing workbenches are open systems because testing needs are organisation-specific  Difficult to integrate with closed design and analysis workbenches

CS 240, Prof. Sarwar Slide 46 A testing workbench

CS 240, Prof. Sarwar Slide 47 Tetsing workbench adaptation  Scripts may be developed for user interface simulators and patterns for test data generators  Test outputs may have to be prepared manually for comparison  Special-purpose file comparators may be developed