PPT12: Fixing the Defect Software Testing - CEN 4072

From defect to failure Defect – An error in the program – especially one that can cause an infection and thus a failure. Also known as a bug or fault Infection – An error in the program state – especially one that can cause a failure “ If debugging is the process of removing software bugs, then programming must be the process of putting them in. ” - Edsger Dijkstra 2

From defect to failure Create Defect Infection Infection Propagates Failure A.K.A SOL This chain must be traced and located to get out of the failure state! How a defect becomes a failure 3

From defect to failure 1.Start with the infected value that defines the failure 2.Determine possible origins of the value, following dependences in the source code 3.Using observation check each single origin to determine whether its truly infected or not. Having found the earlier infection, restart at step 2 How to locate the defect? Guaranteed to isolate the infection chain, but it’s tedious. Nobody got time for that! This determines cause/anomalies but can’t detect where the defect is Combination of induction + experimentation with observation = LOCATE AND ELIMATE THAT BUG 4

Techniques for locating defects Infections: Dependences: Causes: Anomalies: Code smells: 5

Techniques for locating defects How to integrate all these techniques? First determine what technique to focus on Later illustrations will be shown using the techniques 6

Focusing on errors These techniques help focus on specific origins Assertions – Ensure data sanity over long moment in time and a wide range in space. Any failing assertion by definition signals an infection Anomalies – Aspects of the execution whose properties are correlated with failure, such as coverage or dynamic invariants Causes – Aspects of the execution that are not only correlated with failure but actually cause the failure, as experimentally proven 7

Focusing on errors These techniques help focus on finding the defect Infections – If you already know that some origin is faulty – from a failing assertion or via observation - focus on this one first and check whether the infection cause the failure Causes – If delta debugging or any other form of experimentation has highlighted some state or input as a failure cause, focus on these causes and check whether they are infected Anomalies – Of all possible origins those that are associated with anomalies are more likely to contain errors Code smells – If you have determined code smells I you program and if one of these code smells is a possible sign origin Dependences – Anything that is not in the backward slice of the infected state cannot possibly have caused the infection. Of all possible origins, check the backward slice for infections, starting with the closest statement 8

Techniques: dependences Determine the possible origins of the infect value, following dependences in the source code 9

Techniques: observation Check each single origin to determine whether it is infect or not 10

Techniques: assertion Use assertion to rule out possible infection origins – simply because assertion guarantees that the state is sane 11

Techniques: anomaly Still have a choice between two origins. Observing the one that is abnormal reveals an infection 12

Techniques: cause transition Found a cause transition – which is a statement that causes the failure. Real defect and infection chain – due to having the origin being sane and the target being infected 13

Techniques: ordering The order in which one should use the techniques starting with Infections Infections Causes Anomalies Code smells Dependences The rules constitute the “Focus on likely origins” step in the TRAFFIC strategy. Each potential origin must then be verified whether it is infect or not, and repeat the process for infected 14

Techniques: ordering Complexity challenge If a state is sane, then one does not need to consider earlier states. Focusing on the true problems creates time complexity of tracing the pesky bug faster! 15

Is the error a cause? Being on the wrong track is dangerous. Replacing its value with a non-suspicious ones and checking whether the failure no longer occurs ensures that the origin actually causes the error If a outputs 0, it is assumed that foo is infected If a changes to 1 the program still outputs 0 Then it is known that a does not cause the output 16

Is the cause an error? A loop adds up the balance for a specific account Let’s say some people did not get their saving bonus, and Steve is counting on it You have failed Steve he was account 891. Only because the symptom was fixed rather than the cause Making the symptom disappear is not an acceptable solution as it will haunt you later Unless you didn’t like Steve 17

Debugging into existence No clue why this program fails. The boundaries are off within the loop, so replacing size with size - 1. Sounds good right? I get desired outputs and that fixed the problem! Yes! This case is known as ignorant surgery Later the program will blow up and introduce a new defect with the "fix“ 18

19

Devil’s guide to debugging Find the defect by guessing. This includes: Scatter debugging statements throughout the program Try changing code until something works Don’t back up old versions of the code Don’t bother understand what the program should do Don’t waste time understanding the problem. Most problems are trivial anyway 20

Devil’s guide to debugging Use the most obvious fix. Just fix what you see: Why bother going all the way through compute()? x = compute(y); /* compute () doesn’t work for y== 17; so fix it */ if (y == 17) {x = 25.15; } 21

22

Correcting the defect The moment. You did all the steps: tracked the bug down, have complete understanding what is going on and reproduced the failure. The correction is applied Not done yet! Must take care of four problems “ Beware of bugs in the above code; I have only proved it correct, not tried it. ” - Donald Knuth 23

Correcting the defect Does the failure no longer occur? If the failure is done you are a hero If it’s not gone: Being wrong about a correction should Leave you astonished Cause self-doubt, personal reevaluation, and deep soul searching Happen rarely Bring back old code because the defect might be somewhere else 24

Correcting the defect Did the correction introduce new problems? Make sure this does not introduce new problems Have corrections peer reviewed Have a regression test to detect unwanted behavior changes Check each correction individually Introducing new problems is a certain pitfall, and temporarily fixing is not the way to go 25

Correcting the defect Was the same mistake made elsewhere? Check for other defects caused by the same mistake Other code from the same developer Code involving the same APIs Refactoring code helps preventing similar mistakes 26

Correcting the defect Did I commit the change? Be sure to commit your changes to systems The version control system The bug tracking system This will save you the hassle of making the code work after doing something very small. Knowledge of the past is power! 27

Workarounds Defect is not difficult but correcting the defect might be reasons why it might be impossible Unable to change: The program in question cannot be changed – for instance, because it is supplied by a third party and its source code is not available Risks: The correction induces huge risks – for instead because it implies large changes throughout the system Flaw: The problem is not in the code, but overall design – that is the system design must undergo a major revision 28

Workarounds Some cases might need a workaround instead of correction in order to not cause failure A workaround is Not a permanent solution Specific to the situation at hand Tends to reintroduce the failure again Always keep track of the problem! 29

Famous workarounds Spam filters: used to solve a flaw in the system. The proper solution would be to redesign the entire system. Virus scanners: used to solve a flaw of some operating systems. The proper solution would be to assign users limited rights and ask them for proper authorization before a change. Date windowing: used to the inability of many legacy systems to deal with four- digit years. The proper solution would be to adapt the legacy systems. 30

Learning from mistakes Straight forward suggestions for ensuring that problems will not resurface Improve quality assurance: problem will not be missed in the future – and can be fixed more quickly should it occur again Improve your test suite: the test suite failed to detect the problem. Extend the suite so does not happen again (automated test) Set up assertions: inserted assertions that helped tracked the infection should be kept in the code until active production code is finished 31

Learning from mistakes More Straight forward suggestions Improve the software process: If the wrong version goes out or not tested enough. Important parts were not reviewed. It’s the process. Achieve perfection by always thinking about how to improve process Improve your analysis tools: verify whether common tools could have detected the defect early. Tools like system assertions, and tools that detect code smells 32

Facts on fixes In the ECLIPSE and MOZILLA projects, about 30 to 40% of all changes are fixes Fixes are typically two to three times small than other changes Fixes are more likely to induce failures than other changes Only 4% of one-line changes introduce new errors in the code A module that is one year older than another module has 30% fewer errors Newly written code is 2.5 times as defect prone as old code 33

Sources Images: Quotes: Book: Zeller, Andreas. Why Programs Fail: A Guide to Systematic Debugging Amsterdam: Elsevier/Morgan Kaufmann, Print. Slides: Zeller, Andreas. "Fixing the Defect." Why Programs Fail: A Guide to Systematic Debugging - Andreas Zeller - Debugging Book. N.p., n.d. Web. 19 Nov