Anti-patterns in Search-based Program Repair Shin Hwei Tan* Hiroaki Yoshida+ Mukul R. Prasad+ Abhik Roychoudhury* National University of Singapore* Fujitsu Laboratories of America, Inc.+ **Part of this work was conducted during at an internship at Fujitsu Laboratories of America

What is automated program repair? BUG! Given failing Test T , buggy program P Fault localization- Where to fix? Patch Generation - How to fix? Patch Validation – Are all tests passing?

Previous work on Search-Based Program Repair GenProg [ICSE '12] PAR [ICSE '13] SPR [FSE '15] Search Genetic Programming Improved version for adaptive equivalent search (AE) Evolutionary Algorithm Specialized Search with condition synthesis Operators Mutations & crossovers Patterns learned from Human Patches Patterns with and without abstract condition Program C Programs Java Programs

Search-Based Program Repair All Tests Pass Final Patch Candidate Patches Search-Based Repair Tools Patch Generation Tests contains at least one failing test Tests Fail How do the tests look like? Patch Evaluation How do the patches look like?

Search-Based Program Repair $command $argument1 $argument2 RETVAL=$? [ $RETVAL -eq 0 ] && echo Success [ $RETVAL -ne 0 ] && echo Failure Test Script Check exit status of command Non-zero exit status denotes test failure exit(-2); Candidate Patches Tests Patch Evaluation

Repair patterns from human patches Automatic Program Repair Introduce Conditional Control-Flow Loosen Condition Insert Initialization …

Repair patterns from human patches Automatic Program Repair Introduce Conditional Control-Flow Loosen Condition Insert Initialization …

Repair patterns from human patches Automatic Program Repair Conditional Control Flow: +if(a) + return b; int foo(){ + if(input1) + return(out1) //compute something …} Anti-patterns Set of generic forbidden transformations that can be enforced on top of any search-based repair tool.

Prevalence of Anti-patterns Anti-patterns are prevalent in automatically generated patches

Problem: Weak Oracle Statements like exit call/assertions serve as test proxies Test proxies should not be randomly manipulated $command $argument1 $argument2 RETVAL=$? [ $RETVAL -eq 0 ] && echo Success [ $RETVAL -ne 0 ] && echo Failure Failing Test Script Test outcome determined by exit status A1: Anti-delete CFG exit node Remove return statements, exit calls, functions with the word “error”, assertions. static void BadPPM(char* file) { fprintf(stderr, "%s: Not a PPM file.\n", file); - exit(-2); }

Problem: Inadequate Test Coverage Repair tools allow removal of code as long as all test passes Statements are mistakenly considered as redundant code Anti-patterns: A2: Anti-delete Control Statement A3: Anti-delete Single-statement CFG A4: Anti-delete Set-Before-If A2: Anti-delete Control Statement Remove control statements (e.g., if-statements, switch-statements, loops). call_result = call_user_function_ex(...); - if (call_result == SUCCESS && ...) { - if (SUCCESS == statbuf_from_array(...)) - ret = 0; - } else if (call_result == FAILURE) {…

Problem: Inadequate Test Coverage int methodA (int x, int y) { int z = x/y; return x*y; } Consider redundant code since x/y is computed but never used Could be removed by dead-code elimination But removing this line eliminates potential division-by-zero error, hence removal could cause different output Statements may be considered redundant code to test-driven repair tools Test suite with inadequate coverage allows removal of code as long as all test passes Anti-patterns: A2: Anti-delete Control Statement A3: Anti-delete Single-statement CFG A4: Anti-delete Set-Before-If

Problem: Inadequate Test Coverage Anti-pattern: A2 Deletion of CFG node 1 2 3 5 4 7 6 8 PASS FAIL A2: Anti-delete Control Statement Remove control statements (e.g., if-statements, switch-statements, loops). call_result = call_user_function_ex(...); - if (call_result == SUCCESS && ...) { - if (SUCCESS == statbuf_from_array(...)) - ret = 0; - } else if (call_result == FAILURE) {…

Problem: Inadequate Test Coverage Original CFG 1 2 3 5 4 7 6 8 PASS FAIL Empty CFG node 1 2 3 5 7 6 8 PASS FAIL Anti-pattern: A3 A3: Anti-delete Single-statement CFG Remove the statement in CFG node with only one statement fail:{ - ret = 1; }

Problem: Mask Existing Vulnerabilities Existing vulnerabilities can be masked by removing branches via implicit data-flow A4: Anti-delete Set-Before-If Remove variable definition if the variable is used in subsequent if-statement. - tmp = EstimateStripByteCounts(tif, dir, dircount)<0; if(tmp!=0) goto bad;

Problem: Non-termination Automatically generated patches may incorrectly removes loop update Cause infinite loop A5:Anti-delete Loop-Counter Update Remove assignment statement A inside loop L if: 𝑉𝑎𝑟 𝑖𝑛 𝑇𝑒𝑟𝑚𝑖𝑛𝑎𝑡𝑖𝑜𝑛 𝐶𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 𝑜𝑓 𝐿 ∩ {𝑉𝑎𝑟 𝑖𝑛 𝐿𝐻𝑆 𝑜𝑓 𝑎𝑠𝑠𝑖𝑔𝑛𝑚𝑒𝑛𝑡 𝐴}=∅ while( x> 5) - x++;

Problem: Trivial Patch Trivial patch – patch that insert return-statements based on expected output Ex: +if(test1) + return out1 A6: Anti-append Early Exit Insert return/goto statement at any location except for after the last statement in a CFG node. + if ((type != 0)) + return; zend_error((1<<3L),"Uninitialized string offset:",...);

Problem: Functionality Removal Removes functionality by inserting T/F A7: Anti-append Trivial Conditions Insert trivial condition. A condition is trivial if and only if it is: True/False Constant Tautology/Contradiction in expression (e.g., if(x || y || !y)) Static analysis (e.g., if(x || y != 0), y is initialized) - if ((fmap[j].key != format->ptr[i + 1])) + if ((fmap[j].key != format->ptr[i + 1]) && !(1)) continue;

Integrating Anti-patterns Final Patch Search-Based Repair Tools Patch Generation Tests Fail All Tests Pass Candidate Patches Tests contains at least one failing test YES NO Is Anti-pattern? Patch Evaluation

Experimental Setup Corebench benchmark GenProg benchmark Subjects Description kLoC Tests coreutils File, Shell and Text manipulation Utility 83.1 4772 findutils Directory Searching Utility 18.0 1054 grep Pattern Matching Utility 9.4 1582 make Program executable generation utilities 35.3 528 php Programming Language 1046 8471 libtiff Image Processing Library 77 78 python 407 35 gmp Math Library 145 146 gzip Data Compression Utility 491 12 wireshark Network Packet Analyzer 2814 63 fbc Compiler 97 773 lighthttpd Web Server 62 295 Corebench benchmark GenProg benchmark Evaluation on: GenProg AE – Uses Adaptive Equivalent Search SPR – Uses staged repair algorithm with repair patterns dGenProg – GenProg without deletion operator

Evaluation of Anti-patterns RQ1: How do anti-patterns affect the quality of patches generated by search-based program repair tools? RQ2: How many nonsensical patches can our anti-patterns eliminate to reduce manual inspection costs? RQ3: When our modified tools produce the same patch, what is the speedup that we achieve? RQ4: How does the use of anti-patterns compare to an approach that simply prohibits deletion?

Measuring Patch Quality (RQ1) Human generated patch: H Patch generated by Tool: A Original Repair Tool: To Modified Repair Tool: Tm Localizes Correct Line. A and H modify the same line Localizes Correct Function but Incorrect Line. A and H modify statements within the same function. Removes Less Functionality. A removes or skips over fewer lines of code from original program.

Measuring Patch Quality (RQ1) Human generated patch: H Patch generated by Tool: A Original Repair Tool: To Modified Repair Tool: Tm Same Patch. To and Tm generate same repair. Localizes Correct Line. A and H modify the same line Localizes Correct Function but Incorrect Line. A and H modify statements within the same function. Removes Less Functionality. A removes or skips over fewer lines of code from original program. No Repair. To generates a repair but Tm produce no patch.

RQ1: Patch Quality mGenProg outperforms Genprog in: Localizes Correct Line Localizes Correct Function but Incorrect Line Removes Less Functionality Similar improvement in mSPR

RQ4: Comparison with GenProg without deletion (dGenProg) mGenProg outperforms dGenprog in: Localizes Correct Line Localizes Correct Function but Incorrect Line mGenProg removes less functionality than GenProg without deletion! dGenProg removes functionality via insertions of return/goto statements

RQ1: Patch Quality Same Patch: 26 Localizes Correct Line: 1 vs. 15 Localizes Correct Function but Incorrect Line: 15 vs. 12 Removes Less Functionality: 19 Anti-patterns produce better patches in terms of fix localization and removal of functionality in mGenProg Same Patch: 42 Localizes Correct Line: 14 vs. 8 Removes Less Functionality: 10 Anti-patterns produce better patches in terms of fix localization and removal of functionality in mSPR

RQ1: Patch Quality GenProg vs. mGenProg Same Patch: 26 Localizes Correct Line: 1 vs. 15 Localizes Correct Function but Incorrect Line: 15 vs. 12 Removes Less Functionality: 19 Anti-patterns produce better patches in terms of fix localization and removal of functionality in mGenProg

RQ1: Patch Quality SPR vs. mSPR Same Patch: 42 Localizes Correct Line: 14 vs. 8 Localizes Correct Function but Incorrect Line: 15 vs. 12 Removes Less Functionality: 10 Anti-patterns produce better patches in terms of fix localization and removal of functionality in mSPR

RQ2: Reducing Manual Inspection Cost Due to batch compilation, SPR may produce multiple patches Overall, SPR generates 87 patches, mSPR only generates 54 patches. All 33 additional patches are plausible but incorrect patches. Anti-patterns reduce manual inspection cost by eliminating nonsensical repairs.

RQ3: Speedup mGenProg’s Average Repair Time Speedup: 1.42x Repair Space Reduction (%) = 1-( 𝑀𝑜𝑑𝑖𝑓𝑖𝑒𝑑 𝑇𝑜𝑡𝑎𝑙 𝑅𝑒𝑝𝑎𝑖𝑟 𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠 𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑇𝑜𝑡𝑎𝑙 𝑅𝑒𝑝𝑎𝑖𝑟 𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠 )*100 Repair Time Speedup = 𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑅𝑒𝑝𝑎𝑖𝑟 𝑇𝑖𝑚𝑒 Modified Repair Time mGenProg’s Average Repair Time Speedup: 1.42x mSPR’s Average Repair Time Speedup: 1.69x Anti-patterns reduce overall repair time via repair space pruning.

Conclusions and Future Work Enforcing anti-patterns leads to patches that delete less functionality and better fix localization. Tools integrated with anti-patterns generate patches faster due to repair space reduction. Future Work Anti-patterns as specification for guiding repair Anti-patterns as selected “code smells” Tools integrated with anti-patterns can be used as fix localization tools Adapt anti-patterns to other search-based software engineering activities (e.g., specific code anti-patterns identifying energy hot-spots)