Software Testing Sudipto Ghosh CS 406 Fall 99(Also used later) November 2, 1999

11/02/99CS 406 Testing2 Learning Objectives Testing What is testing? Why test? How testing increase our confidence in program correctness? When to test? What to test? How to test? Different types of testing How to assess test adequacy?

11/02/99CS 406 Testing3 Testing The act of checking if a part or product performs as expected. Why do we test? Check if there are any errors? Increase confidence in the “correctness”

11/02/99CS 406 Testing4 Testing When should we test? Separate phase for testing? OR At the end of every phase? OR During each phase, simultaneously with all development and maintenance activities

11/02/99CS 406 Testing5 What can be tested? All products generated during lifecycle PhaseProductHow to test? SpecificationReq. docScenario construction, simulations DesignDesign docSimulations ImplementationModulesRun against test cases IntegrationSystemRun against test cases MaintenanceSystemRegression testing

11/02/99CS 406 Testing6 Our focus Focus: Testing programs Programs may be sub- systems or complete systems Programs are written in a programming language Explore techniques and tools for testing programs

11/02/99CS 406 Testing7 Verification, Validation, Testing Verification: Process of determining whether a phase has been correctly carried out Are we building the product right? (Boehm) Validation: Intensive evaluation process that takes place just before the product is delivered to the client Are we building the right product? (Boehm) V & VV & V denotes testing (Verification: mathematically prove program correctness)

11/02/99CS 406 Testing8 Problems with formal verification Adequate mathematical training Economic viability of proofs Proving hard for non-trivial products Correctness of the “prover” itself Finding input, output specifications, loop invariants Wrong specifications

11/02/99CS 406 Testing9 Terminology Program - Sort Collection of functions (C) or classes (Java) Specification Input: p - array of n integers, n>0 Output: q - array of n integers such that q[0]  q[1]  …  q[n] Elements in q are a permutation of elements in p, which are unchanged Description of requirements of a program

11/02/99CS 406 Testing10 Tests Test input (Test case) A set of values given as input to a program Includes environment variables {2, 3, 6, 5, 4} Test set A set of test cases { {0}, {9, 8, 7, 6, 5}, {1, 3, 4, 5}, {2, 1, 2, 3} }

11/02/99CS 406 Testing11 Oracle Function that determines whether or not the results of executing a program under test is as per the program’s specifications Software Under Test Oracle Test Cases output Failure? Success? Compare Problems Correctness of Oracle Correctness of specs Generation of Oracle Need more formal specs

11/02/99CS 406 Testing12 Correctness Program Correctness A program P is considered with respect to a specification S, if and only if: For each valid input, the output of P is in accordance with the specification S What if the specifications are themselves incorrect?

11/02/99CS 406 Testing13 Errors, defects, faults Often used interchangeably. Error: Mistake made by programmer. Human action that results in the software containing a fault/defect. Defect / Fault: Manifestation of the error in a program. Condition that causes the system to fail. Synonymous with bug

11/02/99CS 406 Testing14 Failure Incorrect program behavior due to a fault in the program. Failure can be determined only with respect to a set of requirement specs. For failure to occur, the testing of the program should force the erroneous portion of the program to be executed.

11/02/99CS 406 Testing15 Errors and failure Inputs Outputs Error revealing inputs cause failures Erroneous outputs indicate failures Program

11/02/99CS 406 Testing16 Debugging Suppose that a failure is detected during the testing of P. The process of finding and removing the cause of this failure is known as debugging. Testing usually leads to debugging. Testing and debugging usually happen in a cycle.

11/02/99CS 406 Testing17 Test-debug cycle Test Debug Done Failure? Testing complete? Yes No

11/02/99CS 406 Testing18 Software quality Extent to which the product satisfies its specifications. Get the software to function correctly. SQA group is in charge of ensuring correctness. Testing Develop various standards to which the software must conform Establish monitoring procedures for assuring compliance with those standards

11/02/99CS 406 Testing19 Nonexecution-Based Testing Person creating a product should not be the only one responsible for reviewing it. A document is is checked by a team of software professionals with a range of skills. Increases chances of finding a fault. Types of reviews Walkthroughs Inspections

11/02/99CS 406 Testing20 Walkthroughs TEAM Client representativeRep of team from next phase Rep from SQA group Rep from Specs teamManager from Specs team Reviewer prepares two lists: Items that the reviewer does not understand Items that the reviewer believes are incorrect

11/02/99CS 406 Testing21 Managing walkthroughs Distribute material for walkthrough in advance. Includes senior technical staff. Chaired by the SQA representative. Task is to record fault for later correction Not much time to fix it during walkthrough Other individuals are trained to fix it better Cost of 1 team vs cost of 1 person Not all faults need to be fixed as not all “faults” flagged are incorrect

11/02/99CS 406 Testing22 Managing walkthroughs (contd.) Two ways of doing walkthroughs Participant driven Present lists of unclear items and incorrect items Rep from specs team responds to each query Document driven Person responsible for document walks the participants through the document Reviewers interrupt with prepared comments or comments triggered by the presentation Interactive process Not to be used for the evaluation of participants

11/02/99CS 406 Testing23 Inspections Proposed by Fagan for testing designs code An inspection goes beyond a walkthrough Five formal stages Stage 1 - Overview Overview document (specs/design/code/ plan) to be prepared by person responsible for producing the product. Document is distributed to participants.

11/02/99CS 406 Testing24 Inspections (contd.) Stage 2 - Preparation Understand the document in detail. List of fault types found in inspections ranked by frequency used for concentrating efforts. Stage 3 - Inspection Walk through the document and ensure that Each item is covered Every branch is taken at least once Find faults and document them (don’t correct) Leader (moderator) produces a written report

11/02/99CS 406 Testing25 Inspections (contd.) Stage 4 - Rework Resolve all faults and problems Stage 5 - Follow-up Moderator must ensure that every issue has been resolved in some way Team Moderator-manager, leader of inspection team Designer - team responsible for current phase Implementer - team responsible for next phase Tester - preferably from SQA team

11/02/99CS 406 Testing26 What to look for in inspections Is each item in a specs doc adequately and correctly addressed? Do actual and formal parameters match? Error handling mechanisms identified? Design compatible with hardware resources? What about with software resources?

11/02/99CS 406 Testing27 What to record in inspections Record fault statistics Categorize by severity, fault type Compare # faults with average # faults in same stage of development Find disproportionate # in some modules, then begin checking other modules Too many faults => redesign the module Information on fault types will help in code inspection in the same module

11/02/99CS 406 Testing28 Pros and cons of inspection High number of faults found even before testing (design and code inspections) Higher programmer productivity, less time on module testing Fewer faults found in product that was inspected before If faults are detected early in the process there is a huge savings What if walkthroughs are used for performance appraisal?

11/02/99CS 406 Testing29 Comparison of inspection and walkthroughs Inspection Checklist used to find faults Five step process Formalized procedure in each step Longer time Walkthrough No checklist used Two step process No formalized procedure in the steps Shorter time

11/02/99CS 406 Testing30 Strengths and weaknesses of reviews Effective way of detecting faults Faults detected early in development Effectiveness reduced if software process is inadequate Large scale software hard to review if it consists of smaller largely independent units (OO) Need access to complete, updated documents from previous phase

11/02/99CS 406 Testing31 Testing and code inspection Code inspection is generally considered complimentary to testing Neither is more important than the other One is not likely to replace testing by code inspection or by formal verification

11/02/99CS 406 Testing32 Program testing Testing is a demonstration that faults are not present. Program testing can be an effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence. - Dijkstra

11/02/99CS 406 Testing33 Execution-based testing Execution-based testing is a process of inferring certain behavioral properties of a product, based, in part, on the results of executing the product in a known environment with selected inputs. Depends on environment and inputs How well do we know the environment? How much control do we have over test inputs? Real time systems

11/02/99CS 406 Testing34 Utility Extent to which a user’s needs are met when a correct product is used under conditions permitted by its specs. How easy is the product to use? Does it perform useful functions? Is it cost-effective?

11/02/99CS 406 Testing35 Reliability Probability of failure-free operation of a product for a given time duration How often does the product fail? mean time between failures How bad are the effects? How long does it take to repair it? mean time to repair Failure behavior controlled by Number of faults Operational profile of execution

11/02/99CS 406 Testing36 Robustness How well does the product behave with range of operating conditions possibility of unacceptable results with valid input? possibility of unacceptable results with invalid input? Should not crash even when not used under permissible conditions

11/02/99CS 406 Testing37 Performance To what extent does the product meet its requirements with regard to response time space requirements What if there are too many clients/ processes, etc...

11/02/99CS 406 Testing38 Levels of testing Unit testing Integration testing System testing Acceptance testing

11/02/99CS 406 Testing39 Test plan Test unit specification Features to be tested Approach for testing Test deliverables Schedule Personnel allocation

11/02/99CS 406 Testing40 References Textbook Scach - Classical and Object-Oriented Software Engineering Other books Beizer - Software Testing Techniques Beizer - Black-Box Testing: Techniques for Functional Testing of Software and Systems Marick - The Craft of Software Testing Myers - The Art of Software Testing