1. Csi565: Theory and Practice of Software Testing Introduction Spring 2009

2. Software Processes  Common fundamental activities to all software processes  Software Specification  The functionality of the software and constraints on its operations must be defined  Software design and implementation  The process of converting specification into executable system  Software validation  The software must be validated to ensure that it does what the customer wants  Software evolution  The software must evolve to meet changing customer need

3. Software specification  AKA requirement engineering (RE)  The most critical stage of the software process  Need to understand/define what services are required from the system

4. Phases in RE  Consists of the following sub-phases  Feasibility study  Requirement elicitation and analysis  Requirements specification  Requirements validation

5. Software design and implementation  The software to meet the specification must be constructed  Design process involves  Architectural design  Abstract specification  Interface design  Component design  Data structure design  Algorithm design

6. Implementation (coding)  Coding or programming is a personal activity does not follow a general process  Some programmer starts with well-understood components  Some takes starts with ill-understood components  Programmer perform testing to reveal program defect  Programmer debug the defect by locating and correcting these defacts

7. Software Validation  AKA verification and validation (V&V)  Shows a system conforms to its specification (i.e., system fulfill the customer expectations)  Heavily depends on checking using  Review  Inspections  Formal methods

8. Software evolution  The flexibility of software is one of the main reasons why more and more software is being incorporated in large, complex systems  Concerns with modifying existing software systems to meet new requirements  Normal approach to software development

9. Testing to check the quality of something by performing experiments in controlled way

10. Some definitions Myers“Testing is the process of executing a program with the intent of finding errors.” [ISO]“Testing is a technical operation that consists of the determination of one or more characteristics of a given product, process, or service according to a specified procedure.” [Pol]“Testing is a process of planning, preparation and measurement aimed at establishing the characteristics of an information system and demonstrating the difference between the actual and the required status.”

11. More on Software testing Software testing is a technical process, performed by executing / experimenting with a product, in a controlled environment, following a specified procedure, with the intent of measuring one or more characteristics / quality of the software product by demonstrating the deviation of the actual status of the product from the required status / specification

12. About testing Testing is not:  static analysis  reviewing or inspection  walk-through  debugging  bug fixing  auditing  quality control  validation of the specification But testing:  is one of the instruments for measuring quality  increases value of product by establishing confidence in its quality  helps in assessing risk of putting product into operation

13. Problems of Testing  Infinity of testing:  too many possible input combinations -- infinite breadth  too many possible input sequences -- infinite depth  too many invalid and unexpected inputs  Exhaustive testing never possible:  when to stop testing ?  how to invent effective and efficient test cases with high probability of detecting errors ?  Optimization problem of testing yield and invested effort  usually stop when time is over......

14. Challenges  No realistic reliability models for software  How to measure the quality of a test suite ?  Test automation  test activities can be boring and require precision  in particular regression testing  difficult  Bad specification or no specification at all  what do you test ?  Many operating environments and contexts

15. Testing Activities  Test generation  analysis of system under test, its specifications, its environment and its interfaces  determination of test strategy  construction and specification of set of test cases  Test execution  implementation of means for execution of specified tests  execution of specified tests  analysis and verdict assignment for executed tests  Test organization  management and planning of test process  allocation of resources  testware management and consolidation for regression testing

16. Basic defintions:1 Quality :Characteristics of a product leading to satisfying requirements, needs, wishes Testing:Measuring the quality of a product; obtaining confidence in the quality of product

17. Basic Definitions:2  Error  AKA bugs  Mistakes made by people (programmers)  Fault  AKA defects  The end result of an error  Occurs during the execution of software and results in an incorrect state that may or may not lead to a failure  Failure  Occurs when a fault (or faulty code) is executed  Incident  Occurs when failure happens

18. Basic definitions: 3  The essence of software testing is to determine a set of test cases fo rthe item being tested  Test cases  Has an identify  Associated with a program behavior (or featute)  Has a set of inputs, and a list of expected outputs

19. Typical Test Case Information TEST Case ID Objective Pre-Conditions Post-Condition Expected Outputs Inputs Execution History (date, result, version, Run by)

20. Why are we Testing ?  Testing as measurement for software quality  Software quality as a measure to predict future operation :  reliability, dependability, MTBF  expected value, expected costs, probability of failure  Risk  probability of failure  expected cost of failure system test acceptance test specbuild deliver exploit & usedestroyaccept quality ? test it !

21. Type of testing  Many different types and sorts of testing:  functional testing, acceptance testing, duration testing,  performance testing, interoperability testing, unit testing,  black-box testing, white-box testing,  regression testing, reliability testing, usability testing,  portability testing, security testing, compliance testing,  recovery testing, integration testing, factory test,  robustness testing, stress testing, conformance testing,  developer testing, acceptance, production testing,  module testing, system testing, alpha test, beta test  third-party testing, specification-based testing,

22. Types of Testing unit integration system performance robustness functional behaviour white boxblack box Level of detail Accessibility Characteristics usability reliability module security

23. implementation code detailed design requirements Specification (SA) acceptance test system test module test unit test Testing in the Development Process : The V-Model

24. Verification is only as good as the validity of the model on which it is based Verification vs. Testing Verification :  formal manipulation  prove properties  performed on model Testing :  experimentation  show error  Performed on actual system Testing can only show the presence of errors, not their absence

25. The objective of of testing  To find failures, or to show correct execution that satisfies the users expectations within the budget, schedule, and resource constraints

26. The testing process  Component testing  Testing of individual program components  Usually the responsibility of the component developer (except sometimes for critical systems)  Tests are derived from the developer’s experience

27. Integration testing  Integration testing  Testing of groups of components integrated to create a system or sub-system  The responsibility of an independent testing team  Tests are based on a system specification

28. Testing phases

29. Defect testing  The goal of defect testing is to discover defects in programs  A successful defect test is a test which causes a program to behave in an anomalous way  Tests show the presence not the absence of defects

30. Testing priorities  Only exhaustive testing can show a program is free from defects.  exhaustive testing is impossible to perform  Tests should exercise a system's capabilities rather than its components  Testing existing capabilities is more important than testing new capabilities  Testing typical/routines cases is more important than very specific cases (i.e., boundary value)

31. The defect testing process

32. Black-box testing  An approach to testing where the program is considered as a ‘black-box’  The program test cases are based on the system specification  Test planning can begin early in the software process

33. Black-box testing (e.g., Roger Pressman)

34. Equivalence partitioning  Input data and output results often fall into different classes where all members of a class are related  Each of these classes is an equivalence partition where the program behaves in an equivalent way for each class member  Test cases should be chosen from each partition

35. Equivalence partitioning (Roger Pressman)

36. Equivalence partitioning  Partition system inputs and outputs into ‘equivalence sets’  If input is a 5-digit integer between 10,000 and 99,999, equivalence partitions are 100, 000  Choose test cases at the boundary of these sets  00000, 09999, 10000, 99999, 10001

37. Equivalence partitions

38. Search routine specification procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ; Pre-condition -- the array has at least one element T’FIRST <= T’LAST Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

39. Search routine - input partitions  Inputs which conform to the pre-conditions  Inputs where a pre-condition does not hold  Inputs where the key element is a member of the array  Inputs where the key element is not a member of the array

40. Testing guidelines (sequences)  Test software with sequences which have only a single value  Use sequences of different sizes in different tests  Derive tests so that the first, middle and last elements of the sequence are accessed  Test with sequences of zero length

41. Search routine - input partitions

42. Structural testing  Sometime called white-box testing  Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases  Objective is to exercise all program statements (not all path combinations)

43. White-box testing

44. Binary search (Java)

45. Binary search - equiv. partitions  Pre-conditions satisfied, key element in array  Pre-conditions satisfied, key element not in array  Pre-conditions unsatisfied, key element in array  Pre-conditions unsatisfied, key element not in array  Input array has a single value  Input array has an even number of values  Input array has an odd number of values

46. Binary search equiv. partitions

47. Binary search - test cases

48. Path testing  The objective of path testing is to ensure that the set of test cases is such that each path through the program is executed at least once  The starting point for path testing is a program flow graph that shows nodes representing program decisions and arcs representing the flow of control  Statements with conditions are therefore nodes in the flow graph

49. Program flow graphs  Describes the program control flow. Each branch is shown as a separate path and loops are shown by arrows looping back to the loop condition node  Used as a basis for computing the cyclomatic complexity  Cyclomatic complexity = Number of edges - Number of nodes +2

50. Cyclomatic complexity  The number of tests to test all control statements equals the cyclomatic complexity  Cyclomatic complexity equals number of conditions in a program  Useful if used with care. Does not imply adequacy of testing.  Although all paths are executed, all combinations of paths are not executed

51. Binary search flow graph

52. Independent paths  1, 2, 3, 8, 9  1, 2, 3, 4, 6, 7, 2  1, 2, 3, 4, 5, 7, 2  1, 2, 3, 4, 6, 7, 2, 8, 9  Test cases should be derived so that all of these paths are executed  A dynamic program analyser may be used to check that paths have been executed

53. Integration testing  Tests complete systems or subsystems composed of integrated components  Integration testing should be black-box testing with tests derived from the specification  Main difficulty is localising errors  Incremental integration testing reduces this problem

54. Incremental integration testing

55. Approaches to integration testing  Top-down testing  Start with high-level system and integrate from the top-down replacing individual components by stubs where appropriate  Bottom-up testing  Integrate individual components in levels until the complete system is created  In practice, most integration involves a combination of these strategies

56. Top-down testing

57. Bottom-up testing

58. Testing approaches  Architectural validation  Top-down integration testing is better at discovering errors in the system architecture  System demonstration  Top-down integration testing allows a limited demonstration at an early stage in the development  Test implementation  Often easier with bottom-up integration testing  Test observation  Problems with both approaches. Extra code may be required to observe tests

59. Interface testing  Takes place when modules or sub-systems are integrated to create larger systems  Objectives are to detect faults due to interface errors or invalid assumptions about interfaces  Particularly important for object-oriented development as objects are defined by their interfaces

60. Interface testing

61. Interfaces types  Parameter interfaces  Data passed from one procedure to another  Shared memory interfaces  Block of memory is shared between procedures  Procedural interfaces  Sub-system encapsulates a set of procedures to be called by other sub-systems  Message passing interfaces  Sub-systems request services from other sub-systems

62. Interface errors  Interface misuse  A calling component calls another component and makes an error in its use of its interface e.g. parameters in the wrong order  Interface misunderstanding  A calling component embeds assumptions about the behaviour of the called component which are incorrect  Timing errors  The called and the calling component operate at different speeds and out-of-date information is accessed

63. Interface testing guidelines  Design tests so that parameters to a called procedure are at the extreme ends of their ranges  Always test pointer parameters with null pointers  Design tests which cause the component to fail  Use stress testing in message passing systems  In shared memory systems, vary the order in which components are activated

64. Stress testing  Exercises the system beyond its maximum design load. Stressing the system often causes defects to come to light  Stressing the system test failure behaviour.. Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data  Particularly relevant to distributed systems which can exhibit severe degradation as a network becomes overloaded

65. References/Acknowledgements  Roger Pressman: Software Eng : A Practitioner Approach, 6 th edition