AutoFLox: An Automatic Fault Localizer for Client-Side JavaScript Frolin S. Ocariza, Jr. Karthik Pattabiraman Ali Mesbah University of British Columbia

Web 2.0 Applications: JavaScript  JavaScript: Implementation of ECMAScript standard  Client-Side JavaScript: used to develop web apps  Executes at client’s browser  Web application’s functionality  Dependability problem: Average of 4 errors per web app in the wild [ISSRE11] 2

JavaScript Dependability: Document Object Model (DOM) 3 html bodyhead scriptdivp Text: “Hello world” table trp Element added by JavaScript Element removed by JavaScript

JavaScript Dependability: Document Object Model (DOM) 4 div id: elem JavaScript code: DOM: var x = document.getElementById(“elem”);var x = document.getElementById(“elme”); Inexistent ID Will return null DOM-related JavaScript error

Fault Localization  Web application testing  What to do after we find errors? Need to fix them…  Fault localization: Find the root cause of the error 5

Contributions/Outline 1. Discussion of JavaScript fault localization challenges 2. Automated technique to localize DOM-related JavaScript faults 3. Empirical studies  Highlight prevalence of DOM-related JavaScript errors  Evaluate accuracy and performance of technique 6

1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); Running Example 7 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000);  Goal: Show a fading banner that cycles through four images every 5 seconds Passed with no argument (even though changeBanner needs one argument) bannerID will be set randomly Would return null if bannerID out of range NULL EXCEPTION! setTimeout call

1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); JavaScript Fault Localization: Challenges  Multiple JS execution sequences executing asynchronously 8 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); Sequence 1 Sequence 2

1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); JavaScript Fault Localization: Challenges (2)  DOM-related JS Errors – fault can be in JS code or DOM – find direct DOM access 9 direct DOM access

element = $(“elem”); b = element.getAttribute(“badAttr”) element.innerHTML = “text”; b.value = “newValue”; Scope of Technique  Types of DOM-related JS errors 10 Code-terminating DOM-related JS errors exception Output DOM-related JS errors function changeToBlue(elem) { elem.style.color = “red”; } Wrong colour change

Block Diagram 11 Instrument JS Code Run Web Application Generate Traces Analyze backward slice Extract Relevant Sequence Partition into Sequences Trace file Web Application direct DOM access Trace Collection Phase Trace Analysis Phase

Trace Collection 1. Intercept JS code in web app and instrument each line 2. Run web app – instrumentation will execute for each line that executes 3. Generate a dynamic trace of all executed lines 12 - Trace contains line number, function name, and values of in-scope variables - Error marker also in trace Instrument JS Code Run Web Application Generate Traces Trace file Web Application 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); trace(); 3 changeTimer = setTimeout(changeBanner, 5000); trace(); 4 5 prefix = “banner_”; trace(); 6 currBannerElem = document.getElementById(prefix+currentBannerID);trace(); 7 bannerToChange = document.getElementById(prefix + bannerID); trace(); 8 currBannerElem.removeClassName(“active”); trace(); 9 bannerToChange.addClassName(“active”); trace(); 10 currentBannerID = bannerID; trace(); 11 } 12 currentBannerID = 1; trace(); 13 changeTimer = setTimeout(changeBanner, 5000); trace(); 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); trace(); 3 changeTimer = setTimeout(changeBanner, 5000); trace(); 4 5 prefix = “banner_”; trace(); 6 currBannerElem = document.getElementById(prefix+currentBannerID);trace(); 7 bannerToChange = document.getElementById(prefix + bannerID); trace(); 8 currBannerElem.removeClassName(“active”); trace(); 9 bannerToChange.addClassName(“active”); trace(); 10 currentBannerID = bannerID; trace(); 11 } 12 currentBannerID = 1; trace(); 13 changeTimer = setTimeout(changeBanner, 5000); trace(); Trace Record Prefix: changeBanner:::4 Variables: currentBannerID (global): 1 changeTimer (global): 2 bannerID (local): -11 prefix (local): none currBannerElem (local): none bannerToChange (local): none

Trace Analysis 1. Trace is divided into sequences 2. Relevant sequence – execution sequence that contains error marker 3. Analyze backward slice of last variable that held null to see where null value originated. The line where null value originated is the direct DOM access. 13 Instrument JS Code Run Web Application Generate Traces Analyze backward slice Extract Relevant Sequence Partition into Sequences Trace file direct DOM access Web Application

1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); Sequences.: (1) line2 -> line3 -> line5 -> line6 -> line7 -> line8 -> line9 (2) line12 -> line13 Trace Analysis: Example 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); 14 Relevant Seq.: line12 -> line13 -> line2 -> line3 -> line5 -> line6 -> line7 -> line8 -> line9 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); bannerToChange being assigned return value of DOM access function Error Marker Technique outputs this line as direct DOM access Last variable to take on null value

Implementation of AutoFLox  Trace Collection: Modified versions of existing tools (InvarScope [Groeneveld et al.] and Crawljax [Mesbah et al.])  Trace Analysis: Written from scratch 15

Evaluation: Research Questions  RQ1: How prominent are DOM-related JavaScript errors in web applications?  RQ2: What is the fault localization accuracy of AutoFLox?  RQ3: What is the performance overhead of AutoFLox? 16

Prominence of DOM-related JS Errors  RQ1 Approach: Study bug reports of four web apps (TUDU, TaskFreak, WordPress, and Google)  Only JS-related bugs marked valid or fixed are studied 17

Research Questions  RQ1: How prominent are DOM-related JavaScript errors in web applications?  RQ2: What is the fault localization accuracy of AutoFLox?  RQ3: What is the performance overhead of AutoFLox? 18

Accuracy of AutoFLox  RQ2 Approach: Fault injection experiment on TUDU, TaskFreak, and WordPress  Faults injected: Null exceptions  Mutate the parameters of getElementById and getAttribute methods 19

Accuracy of AutoFLox: Results  WordPress: anonymous functions not supported by AutoFLox at the moment  Direct DOM access in Tumblr also found 20 Web AppTotal Number of Mutations Number of direct DOM accesses identified Percentage identified TaskFreak29 100% TUDU24 100% WordPress % Overall %

Research Questions  How prominent are DOM-related JavaScript errors in web applications?  What is the fault localization accuracy of AutoFLox?  What is the performance overhead of AutoFLox? 21

Performance  RQ3 Approach: Measure trace collection overhead in Tumblr; measure time it takes for trace analysis to complete  Tumblr used because production code more complex than development code  Results  Trace collection incurred 35% overhead  Trace analysis took seconds to complete 22

Conclusion  DOM-related JS errors are prevalent  Created fault localization tool for JS called AutoFLox   Evaluated accuracy and performance of AutoFLox  Future work  Relax simplifying assumptions  Target Output DOM-related JS errors  Automatically fixing DOM-related JS errors 23

Backup Slides

25 Web 2.0 Applications: Document Object Model - Defines elements contained in webpage

Dependability Problems in JavaScript  Big contributing factor: JS interaction with Document Object Model (DOM) 26 DIV id: elem DOMJavaScript Code var x = document.getElementById(“elem”); var x = document.getElementById(“elme”); Will return nullInexistent ID DOM-related JavaScript error

Fault Localization  Web application testing  What to do after we find errors? Need to fix them…  Fault localization: Find the root cause of the error 27 x = document.getElementById(“elme”); changeText(x); ….. function changeText(domElement) { y = domElement; y.innerHTML = ‘New Text’; } exception root cause

Running Example 28 1 function changeBanner(bannerID) { 2 clearTimeout(changeTimer); 3 changeTimer = setTimeout(changeBanner, 5000); 4 5 prefix = “banner_”; 6 currBannerElem = document.getElementById(prefix+currentBannerID); 7 bannerToChange = document.getElementById(prefix + bannerID); 8 currBannerElem.removeClassName(“active”); 9 bannerToChange.addClassName(“active”); 10 currentBannerID = bannerID; 11 } 12 currentBannerID = 1; 13 changeTimer = setTimeout(changeBanner, 5000); Passed with no argument (even though changeBanner needs one argument) bannerID will be set randomly Would return null if bannerID out of range NULL EXCEPTION!

AutoFLox: Scope  Types of DOM-related JS errors  Code-terminating DOM-related JavaScript Errors: Errors that occur when DOM access function returns null, undefined, or incorrect value  Output DOM-related JavaScript Errors: Errors that occur when DOM update function sets DOM element’s property to wrong value 29 AutoFLox is concerned with code-terminating DOM-related JavaScript errors

Limitations  User has to know how to replicate error  User required to specify elements to click during crawling  Multiple JS errors currently not supported 30