Software Testing Strategies

Software Testing Strategy Integration of software test case design methods into a well-planned series of steps Successful software construction Provides a road map that describes The steps to be conducted as part of testing How much effort, time and resources will be required Must incorporate test planning, test case design, test execution, and resultant data collection and evaluation

Software Testing Strategy Should be flexible enough to promote a customized testing approach Must be rigid enough to promote reasonable planning and management tracking as the project progresses Different test methods are beginning to cluster themselves into distinct approaches and philosophies 3

Software Testing What is it? To uncover errors How do we conduct the test? Do we develop a formal plan? Should we test the entire program as a whole or run tests on a small part of it? Should we rerun tests we have already conducted as we add new components to a large system? When should we involve customer? These questions must be answered when we develop a software testing strategy. 4

Software Testing Who does it? Why is it important The strategy is developed by the project manager, software engineers, and testing specialists. Why is it important Testing often accounts for more project effort than any other software engineering activity If conducted haphazardly time is wasted Unnecessary effort is expended Errors sneak through undected 5

Software Testing What are the steps? Begins in the small and progresses to the large Focus on a single component or a small group of related components After components are tested they may be integrated until the complete system is constructed As errors are uncovered, they must be diagnosed and corrected using a process that is called debugging 6

Software Testing What is the work product? A test specification document Defining a plan that describes overall strategy A procedure that defines specific testing steps and the tests that will be conducted How do I ensure that I’ve done it right? By reviewing the Test specification prior to testing Assess the completeness of test cases and testing tasks 7

Software Testing A Strategic Approach Testing is a set of activities that can be planned in advance and conducted systematically Conduct effective formal technical reviews Begin at the component level and work “outward” toward the integration of the entire computer-based system Different testing techniques at different points in time Usually conducted by the software developer and for large projects by an independent test group Testing and debugging are different activities, but debugging must be accommodated in any testing strategy 8

Software Testing Verification and Validation Software testing is often referred to as V&V Verification refers to the set of activities that ensures that software correctly implements a specific function Are we building the product right? Validation is a different set of activities that ensure that the software that has been built is traceable to customer requirements Are we building the right product? 9

Software Testing Verification and Validation Encompass a wide array of Software Quality Assurance (SQA) activities including Formal technical reviews Quality and configuration audits Performance monitoring Simulation Feasibility study Documentation review Database review Algorithm analysis Development, usability, installation testing 10

Software Testing Organizing for Software Testing Psychological point of view The software engineer analyzes, models, and then creates a computer program and its documentation From the software engineers perspective testing is destructive as it tries to break the thing that SE has built There are often a number of misconceptions The developer of the software should no do test Software should be tossed over the wall to strangers who will test it mercilessly Testers get involved with the project only when the testing steps are about to begin 11

Software Testing Organizing for Software Testing The role of the independent test group Remove the inherent problems associated with letting the builder test the thing that has been built Removes the conflict of interest The software engineer, however, does turn the program over to the ITG and walk away; both have to work closely throughout the project The developer must be available to correct uncovered errors 12

Software Testing Strategy for conventional software architecture Unit Testing Concentrates on each unit (i.e. component) of the software as implemented in the source code Integration Testing Focus is on design and the construction of the software architecture Validation Testing Requirements are analysis are validated against the software that has been constructed System Testing Software and other elements are tested as a whole 13

Software Testing Strategy for Object Oriented Architecture Focus on “testing in the small” and work outward toward “testing in the large” Testing in the small changes from an individual module to a class encompassing attributes and operations; implies communication and collaboration A series of regression tests are run to uncover errors due to communication and collaboration between classes 14

Software Testing Strategic Issues Specify product requirements in a quantifiable manner long before testing A good strategy not only looks for errors, but also asses other quality characteristics such as Portability Maintainability Usability State testing objectives explicitly The specific objectives of testing should be stated in measurable terms Test effectiveness The cost to find and fix defects Test coverage 15

Software Testing Strategic Issues Understand the user of the software and develop a profile for each Use-cases that describe the interaction scenario for each class of user can reduce overall testing effort by focusing testing on actual use of the project Build robust software that is designed to test itself Software should be designed in a manner that uses anti-bugging techniques. Software should be capable of diagnosing certain classes of errors. The design should be accommodated automated testing and regression testing 16

Software Testing Strategic Issues Use effective formal technical reviews as a filter prior to testing Reviews can reduce the amount of testing effort that is required to produce high-quality software Develop a continuous improvement approach The testing strategy should be measured. The metrics collected during testing should be used as part of a statistical process control approach for testing 17

Software Testing Test Strategies for Conventional Software There are many strategies for testing software. A software team could wait until the system is fully constructed and then conduct tests on the overall system. At the other extreme software engineer can conduct tests on a daily basis, whenever any part of the system is constructed A testing strategies falls between the two extremes Begin with the testing of individual program units Moving to tests designed to facilitate the integration of the units 18

Software Testing Test Strategies for Conventional Software Unit Testing Focuses verification effort on the smallest unit of software design – the software component or module Integration Testing “If they all work individually, why do you doubt that they’ll work when we put them together?” The problem is “putting them together” – interfacing Data may be lost One module can have an inadvertent, adverse effect on another; sub-function when combined Global data structure can present problems 19

Software Testing Test Strategies for Conventional Software Regression Testing Each time a new module is added as part of integration testing, the software changes New data flow paths are established New I/O and control logic is invoked Regression testing is the re-execution of some subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects Smoke testing An integration testing 20

Software Testing Test Strategies for Conventional Software Smoke testing An integration testing Software components that have been translated into code are integrated into a “build” – data files, libraries, reusable modules, and engineered components that are required to implement one or more product The intent of testing here is to uncover show stoppers – errors that have the highest likelihood of throwing the software project behind schedule The build is integrated with other builds and the entire product is smoke tested daily 21

Software Testing Test Strategies for OBJECT-ORIENTED SOFTWARE Unit Testing An encapsulated class is the focus of unit testing Operations within the class are, however, the smallest unit Integration Testing Thread based testing Integrates a set of classes required to respond to one input or event for the system Used based testing Begins the construction of the system by testing independent classes – use very few if any server classes Testing dependent classes – which use independent classes 22

Software Testing Test Strategies for OBJECT-ORIENTED SOFTWARE Integration Testing Cluster Testing A cluster of collaborating classes (determined by examining the CRC) to uncover errors in collaboration 23

Software Testing Validation Testing At this level the distinction between conventional and object-oriented software disappears Testing focuses on user visible actions and user recognizable output from the system Can be defined in many ways A simple definition is that validation succeeds when software functions in a manner that can be reasonably expected by the customer. Alpha Testing Conducted at the developer site by end-users Beta Testing Conducted at end user sites – the developer is generally not present 24

Software Testing System Testing Ultimately, software is incorporated with other system elements Hardware, people, information A series of system integration and validation tests are conducted A classic system testing problem – finger pointing Software engineers should anticipate potential interfacing problems Design error handling paths that test all information coming from other elements of the system Conduct series of tests to simulate bad data Record the results of testing as evidence Participate in planning and design of systems tests to ensure that software is adequately tested 25

Software Testing System Testing Recovery Testing Is a system test that forces the software to fail in a variety of ways and verifies that recovery is properly performed If automatic recovery Re-initialization Check-pointing mechanisms Data recovery Restart are evaluated for correctness If manual Mean time to repair (MTTR) is evaluated 26

Software Testing Security Testing Stress Testing Performance Testing It verifies that protection mechanisms built into a system will, in fact protect it from improper penetration Stress Testing It executes a system in a manner that demands resources in abnormal quantity, frequency, or volume Performance Testing Are often coupled with stress testing and usually require both hardware and software instrumentation. To measure resource utilization 27