1. Fixing the Defect CEN4072 – Software Testing

2. From Defect to Failure How a defect becomes a failure: 1. The programmer creates a defect 2. The defect causes an infection 3. The infection propagates 4. The infection causes a failure 2Yuriy Zaynulin, 2015

3. How to locate a defect (cont.) 3 Back Yuriy Zaynulin, 2015

4. From Defect to Failure (cont.) How to locate a defect: 1. Identify infected value that defines the failure (pic. [a]) 2. Determine the possible origins of the infected value (pic. [b]) 3. Check every possible origin for infection (pic. [c])  Found the earlier infection? Go to step 2 4 To picture Yuriy Zaynulin, 2015

5. Techniques for Locating Defects  Infections e.g. a failed assertion  Causes e.g. “ a[2] = 0 ” causes the failure  Anomalies e.g. f() executed only in failing run  Code Smells e.g. uninitialized variable  Dependencies e.g. a[2] comes from a[0] 5Yuriy Zaynulin, 2015

6. Techniques for Locating Defects (cont.) Assertions:  ensures data sanity  failing assertion signals an infection  passing assertion needs not be considered Anomalies -  aspects of code correlated with failure Causes -  aspects of execution correlated with failure AND causing the failure 6Yuriy Zaynulin, 2015

7. Techniques: Dependences Dependences :  check the backward slice for infections  start with the closest statements  anything not in the backward slice CANNOT have caused the infection 7 [x] is a dependency of [!]s Yuriy Zaynulin, 2015

8. Techniques: Observation Observation:  allows a programmer to tell (or specify) what is correct and what isn’t correct 8 Observe: transition from SANE to INFECTED Yuriy Zaynulin, 2015

9. Techniques: Ordering 1. Focus on Infections [ if an origin is already known to be faulty, else: ] 2. Focus on Causes [ if delta debugging or experimentation found the cause of a failure, else: ] 3. Focus on Anomalies [ if they are present, else: ] 4. Focus on Code Smells [ if they are present, else: ] 5. Focus on Dependences [ if nothing else worked, examine backward slices; ] 9Yuriy Zaynulin, 2015

10. Is the Error a Cause? Origin is an error, but doesn’t cause the failure? → Wrong track! Example: a = compute_value(); printf("a = %d\n", a);  output is 0  we assume compute_value() produces infection  set a to 1 right before printf(...), still output is 0 → actual problem is with printf(...) 10Yuriy Zaynulin, 2015

11. Is the Cause an Error? (cont.) balance[account] = 0.0; for (int position = 0; position < numPositions; position++) { balance[account] += deposit[position]; } Problem: sum for account #123 is wrong, apply BAD “fix”: if (account == 123) balance[123] += 45.67; Results:  original problem doesn’t show up  new ones might arise  actual cause is unknown → We debugged the code into existence! 11 Back Yuriy Zaynulin, 2015

12. Is the Cause an Error? Origin causes the failure, but not [sure whether it is] an error? → DO NOT debug it into existence! Example: 12 To example Yuriy Zaynulin, 2015

13. 1. Find the defect by guessing:  Scatter debug statements all over  Try changing code until something works  Don’t backup 2. Don’t waste time understanding the problem 3. Use the most obvious fix Example : x = compute(y); // compute() doesn’t work for y == 17, so fix it if (y == 17) x = 25.15; 13Yuriy Zaynulin, 2015

14. Correcting the Defect 14Yuriy Zaynulin, 2015

15. Workaround Not a permanent solution Defect DOES NOT cause a failure anymore Defect DOES still remain → Keep the problem open [in the tracing system] 15Yuriy Zaynulin, 2015

16. Workaround (cont.) When it is appropriate to use: Unable to change  Supplied by third party  Closed source code Risks  Large changes throughout the system needed Flaw  Code is fine, but: Design must undergo major revision 16Yuriy Zaynulin, 2015

17. Workaround (cont.) Famous workarounds:  Spam filters Flaw: anyone can easily forge email sender identity Proper solution: redesign entire e-mail system [costs and risks]  Virus scanners Flaw: OS users have administrator rights, any program can gain full control Proper solution: assign users limited rights and add proper authorization [many good programs assume administrator right for installation] 17Yuriy Zaynulin, 2015

18. Learning From Mistakes Improve test suite  Tests failed to detect the problem → extend the tests. Set up assertions  Keep assertions used in debugging. Add more. Leave assertions active in production code. Improve training  Teach programmers about potential pitfalls and how to avoid them. Improve the software process  Software was not tested fully? Wrong version was shipped? Critical part was not reviewed? Work on production process! Improve analysis tools 18Yuriy Zaynulin, 2015

19. Facts on Fixes 1. In the ECLIPSE and MOZILLA projects, about 30 to 40% of all changes are fixes. 2. Fixes are typically two to three times smaller than other changes. 3. Fixes are more likely to induce failures than other changes. 4. Only 4% of one-line changes introduce new errors in the code. 5. A module that is one year older than another module has 30% fewer errors. 6. Newly written code is 2.5 times as defect prone as old code. Yuriy Zaynulin, 201519

20. Summary To isolate infection: Work backward along infections origins To find most likely origin: Focus on  Failing assertions  Causes  Anomalies  Code smells Each origin must be an infection as well as a cause; Use workaround if correction is too costly or risky Yuriy Zaynulin, 201520

21. Summary (cont.) To correct the defect: know how  Your change breaks the infection chain  Will the failure no longer occur To ensure your correction is successful: check  The failure no longer occurs  No new problems were introduced Yuriy Zaynulin, 201521

22. Sources CADD Manager, “When Workarounds Become Best Practices”, http://www.caddmanager.com/CMB/2010/09/when-workarounds-become-best-practices/http://www.caddmanager.com/CMB/2010/09/when-workarounds-become-best-practices/, 2010 Graves et al., 2000 Mockus, Votta, 2000 Mockus, Weiss, 2000 Ostrand, Weyuker, 2002 Purushothaman, Perry, 2004 Sliwerski et al., 2005 Zeller A., “Why Programs Fail: A Guide to Systematic Debugging”, 2005 Yuriy Zaynulin, 201522