Testing and Inspecting to Ensure High Quality Defects in Ordinary Algorithms Defects in Handling Stress or Unusual Situations

Defects in Ordinary Algorithms Incorrect logical conditions Defect: The logical conditions that govern looping and if- then-else statements are wrongly formulated. Testing strategy: Use equivalence class and boundary testing. Consider as an input each variable used in a rule or logical condition. Break into smaller, individual logical tests. Chapter 10: Testing and Inspecting for High Quality 2 © Lethbridge/Laganière 2001

Example of incorrect logical conditions defect What is the hard-to-find defect in the following code? The landing gear must be deployed whenever the plane is within two minutes from landing or takeoff, or within 2000 feet from the ground. If visibility is less than 1000 feet, then the landing gear must be deployed whenever the plane is within three minutes from landing or lower than 2500 feet If (!landingGearDeployed && (min(now-takeoffTime , estLandTime-now))< (visibility < 1000 ? 180 :120) || relativeAltitude < (visibility < 1000 ? 2500 :2000) ) { throw new LandingGearException(); } Class Exercise! What is the order of evaluation – for starters!! Chapter 10: Testing and Inspecting for High Quality 3 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms Performing a calculation in the wrong part of a control construct Defect: The program performs an action when it should not, or does not perform an action when it should. Typically caused by inappropriately excluding or including the action from a loop or a if construct. Testing strategies: Design tests that execute each loop zero times, exactly once, and more than once. Ensure a Do…While (or While…) is false to begin with Ensure a Perform … Until is true to begin with Remember: Pre-tests and Post Tests Anything that could happen while looping is made to occur on the first, an intermediate, and the last iteration. Chapter 10: Testing and Inspecting for High Quality 4 © Lethbridge/Laganière 2001

Chapter 10: Testing and Inspecting for High Quality Example of Performing a Calculation in the Wrong Part of a Control Construct What is wrong with the following loop?: while( j < maximum) { k=someOperation(j); j++; } If (k == -1) signalAnError(); Chapter 10: Testing and Inspecting for High Quality 5 © Lethbridge/Laganière 2001

Chapter 10: Testing and Inspecting for High Quality What is wrong with the following loop: while( j < maximum) { k=someOperation(j); j++; } If(k == -1) signalAnError(); Must assume j has an acceptable integral value prior to entering loop. Could say: while (j++ < maximum) but this would affect argument (j)! Chapter 10: Testing and Inspecting for High Quality 6 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms 1 Not terminating a loop or recursion Defect: A loop or a recursion does not always terminate, i.e. it is ‘infinite’. Remember in recursion, you have the ‘recursive case’ and the ‘base case’ What do we mean by the ‘recursive case?’ Properties?? What do we mean by the ‘base case?’   Testing strategies: Analyze what causes a repetitive action to be stopped. Run test cases that you anticipate might not be handled correctly. Count on these happening during operations! Does recursive case converge to base case? Chapter 10: Testing and Inspecting for High Quality 7 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms 3 Not setting up the correct preconditions for an algorithm Defect: Preconditions state what must be true before the algorithm is executed. A defect would exist if a program proceeds to do its work, even when the preconditions are not satisfied. Do we really want to ensure a < 14 is true prior to going into the algorithm? while (a<14) { …. ……} such that even if a >= 14, this loop (and perhaps other contents) continues to run appropriately. Testing strategy: Run test cases in which each precondition is not satisfied. See what happens beyond the while-loop. Absolutely! Insert probes (Displays, printf, system.out.println…) Chapter 10: Testing and Inspecting for High Quality 9 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms 4 Not handling null conditions Defect: A null condition is a situation where there normally are one or more data items to process, but sometimes there are none. It is a defect when a program behaves abnormally when a null condition is encountered. How do you test null (or non-numeric) condition in, say, COBOL, for a numeric field? How can it happen? Testing strategy: Brainstorm: determine unusual conditions and run appropriate tests. Always test ‘no inputs’ interactively or via an OPENed file. Always test to see if the Input or Read or … retrieved data was non-null! Always test with only one input; Oftentimes code runs perfect with a single input; Always test with two or more. If code runs correctly with two inputs, it’ll probably run with n inputs (if we are dependent upon checking null or EOF conditions, etc. Chapter 10: Testing and Inspecting for High Quality 10 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms 5 Not handling singleton or non-singleton conditions Defect: A singleton condition occurs when there is normally more than one of something, but sometimes there is only one. (this is a little different than the last case) A non-singleton condition is the inverse. Defects occur when the unusual case is not properly handled. Testing strategy: Brainstorm: determine unusual conditions and run appropriate tests. Can occur with incorrect number of parameters (in some languages). - Can abend or continue, providing null values to unmatched parameters… Think argc and argv. Can easily test. Chapter 10: Testing and Inspecting for High Quality 11 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms 6 Off-by-one errors Defect: A program inappropriately adds or subtracts one. Or loops one too many times or one too few times. This is a particularly common type of defect. Thoughts: Results: often EOF reached; executing same instructions twice; Very easy in C and Java: side effects of a++ and ++a; careful in using a++ or a++ within predicate (condition) of loop as in while (a++ < b) is quite different than while (++a < b) ! Test: if a = 1 and b = 2, which of the two preceding, if any, were true? COBOL: Perform …. Until X >= 25 or ….Until X=25! Java, C++ for (int i = 1; i < 25; i++) ….. ++i or i++ no difference here. Always able to give correct value if ‘i’ within loop”!   What is value of loop control variable when we terminate the loop? Testing strategy: Develop tests in which you verify that the program: computes the correct numerical answer. performs the correct number of iterations. Always display value of loop control variables upon exiting loop!!! Insert probes for counting!!!! Chapter 10: Testing and Inspecting for High Quality 12 © Lethbridge/Laganière 2001

Example of off-by-one defect 7 for (i=1; i<arrayname.length; i++) { /* do something */ } Difference between length and size???? What if arrayname.length = 1? Use Iterators to help eliminate these defects while (iterator.hasNext()) { anOperation(++val); } Nice boolean checks are almost always benign! Chapter 10: Testing and Inspecting for High Quality 13 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms 8 Operator precedence errors Defect: An operator precedence error occurs when a programmer omits needed parentheses, or puts parentheses in the wrong place. Operator precedence errors are often extremely obvious... but can occasionally lie hidden until special conditions arise. E.g. If x*y+z should be x*(y+z) this would be hidden if z =0 and perhaps is normally zero. Testing: In software that computes formulae, run tests that anticipate such defects. Know the operator precedence rules! Don’t parenthesize to death! It shows you don’t know what you are doing! Chapter 10: Testing and Inspecting for High Quality 14 © Lethbridge/Laganière 2001

Defects in Ordinary Algorithms 9 Use of inappropriate standard algorithms Defect: An inappropriate standard algorithm is one that is unnecessarily inefficient or has some other property that is widely recognized as being bad. Can you think of any? How about Order n**2 sorts? How about QuickSort? How about Address Calculation sorts (various kinds)?   Testing strategies: The tester has to know the properties of algorithms and then design tests that will determine whether any undesirable algorithms have been implemented. There is no ‘one size fits all.’ Knowing some of the data characteristics is a big help! Chapter 10: Testing and Inspecting for High Quality 15 © Lethbridge/Laganière 2001

Defects in Handling Stress and Unusual Situations Incompatibility with specific configurations of hardware or software Defect: The system fails if it is run using particular configurations of hardware, operating systems, and external libraries. (How about browsers too?) Testing strategy: Extensively execute the system with all possible configurations that might be encountered by users. For standard software, must anticipate (or restrict) platforms! Chapter 10: Testing and Inspecting for High Quality 16 © Lethbridge/Laganière 2001

Defects in Handling Stress and Unusual Situations 2 Defects in handling peak loads or missing resources Defects: The system does not gracefully handle resource shortage. Resources that might be in short supply include: memory, disk space or network bandwidth, permissions, processors, other peripherals, …. Honeywell: PAT Table Overflow (< 32 files) The program being tested should report problems in a way the user will understand. Testing strategies: Devise a method of denying resources (test missing resources, such as disk drives not available, etc.). Run a very large number of copies of the program being tested, all at the same time. May require some redesign or rethinking of the process! Chapter 10: Testing and Inspecting for High Quality 17 © Lethbridge/Laganière 2001

Defects in Handling Stress and Unusual Situations 3 Inappropriate management of resources Defect: A program uses certain resources but does not make them unavailable when no longer needed. Memory leakage? C – free? Java? Always good? No.  Testing strategy: Run the program intensively in such a way that it uses many resources, relinquishes them and then uses them again repeatedly. Can require some sophisticated testing! Can require heavy loading! Chapter 10: Testing and Inspecting for High Quality 18 © Lethbridge/Laganière 2001

Defects in Handling Stress and Unusual Situations 4 Defects in the process of recovering from a crash Defects: Any system will undergo a sudden failure if its hardware fails, or if its power is turned off. It is a defect if the system is left in an unstable state and hence is unable to fully recover. It is also a defect if a system does not correctly deal with the crashes of related systems. Testing strategies: Kill a program at various times during execution. Try turning the power off; however, operating systems themselves are often intolerant of doing that. Abruptly pre-empt key processes. Chapter 10: Testing and Inspecting for High Quality 19 © Lethbridge/Laganière 2001

Chapter 10: Testing and Inspecting for High Quality Disaster Recovery Earthquakes, Hurricanes, Tornadoes, Floods. What do we do??? Discussion of Disaster Recovery approaches Data – the real valuable corporate asset Backup computing platforms Renting / leasing resources, CPU cycles, disk space, etc. Procedures for evacuation fire, emergency generators, travel. etc. Chapter 10: Testing and Inspecting for High Quality 20 © Lethbridge/Laganière 2001