CrossCheck: Combining Crawling and Differencing to Better Detect Cross-browser Incompatibilities in Web Applications Shauvik Roy Choudhary, Mukul Prasad, Alex Orso Georgia Institute of Technology Fujitsu Labs of America

Move to the Web 2

Multitude of Browsing Environments 3

Georgia Tech Website Mozilla Firefox Internet Explorer 4

IEEE TSE Website 5

Granta Books Website 6 Internet Explorer Mozilla Firefox

Mimic end user’s perception Produce human Consumable reports Manual inspection is expensive Need automated classification ? Modern apps have many dynamic screens DOM differs between browsers 7 Ignore variable elements on webpage 15 Sep :15pm Work with browser security controls

Running Example 8

9

10

Running Example 11

Running Example 12

Running Example 13

Running Example 14 Missing screen transitions 1

Running Example 15 Missing shadow 2 Wrong count of items 3

Running Example 16

Running Example 17

A Tale of Two Techniques 18 WebDiff Screen-level differences (graph matching, computer vision) CrossT Trace-level differences (graph isomorphism)

Goal: To better detect both functional and visual XBIs by combining the two complementary techniques High level view of the approach Web Application CrossCheck Cross-browser Error Report 1. Model Generation 2. Model Comparison 3. Report Generation 19

Screen Transition Graph 1. Model Generation Web Application DOM Tree Screen image + geometries Ajax Crawler (Crawljax) Screen Model Screen image + geometries DOM Tree 20

2a. Trace Level Comparison STG Match Screen Transition Graphs from two browsers using graph isomorphism (both screens and transitions) Output: Pairs of matching screens  Perform Screen-level comparison Unmatched Screens  Report Trace-level issue Unmatched Transitions  Report Trace-level issue STG 21

2a. Trace Level Comparison S0S0 S1S1 S2S2 S1’S1’S2’S2’ 22 S0’S0’

2b. Screen Level Comparison Compare DOMs to find matching elements Use MatchIndex f (xPath, coords, zindex, visibility etc. ) Output: Pairs of matched DOM nodes (corr. screen elements)  for visual comparison Unmatched DOM nodes if any Given a pair of matched screens (from prev. step) 23 Match DOMs Corr. Screen elements

2b. Screen Level Comparison Like-colored (matched) screen-elements compared through visual comparison 24

Machine Learning based Classifier Machine Learning based Classifier YES NO Visual Comparison Extract Features Features Used: Size difference Position difference/ displacement Absolute size Text value difference (based on DOM)  2 –Image distance (between images of 2 elements)  ? Do these screen elements look the same ? 25

Machine Learning based Classifier Training the classifier (offline – single time) Feature Extraction False True Screens with (known) Cross-Browser Differences Set of screen-element comparison instances 26

3. Report Generation (Clustering) 27

Empirical Evaluation RQ1 (Effectiveness) Can CrossCheck identify different kinds of CBDs in real-world web applications and correlate them to identify XBIs? RQ2 (Improvement) How effective is CrossCheck when compared to CrossT and WebDiff? 28

Subjects: Experimental Setup Web AppURLType STDOM Nodes minmaxavg Restaurant Information Organizer Productivity GrantaBooks Publisher DesignTrust Business DivineLife Spiritual SaiBaba Religious Breakaway Sport

Procedure: 1.Used latest versions of Firefox (v7.0.1) and Internet Explorer (v9.0.3) 2.Ran CrossCheck on the subjects to perform the various phases of the technique 3.Manually checked results for false positives and negatives Experimental Setup 30

RQ1: Effectiveness ApplicationTracePositionSizeVisibilityTextAppearanceCBDsXBIs Restaurant Organizer GrantaBooks DesignTrust DivineLife SaiBaba Breakaway TOTAL

RQ1: Effectiveness ApplicationTracePositionSizeVisibilityTextAppearanceCBDsXBIs Restaurant Organizer GrantaBooks DesignTrust DivineLife SaiBaba Breakaway TOTAL

RQ2: Improvement Application CrossCheck CrossTWebDiff Rep.Conf.TraceScreenRep.Conf. Rep.Conf. Restaurant Organizer GrantaBooks DesignTrust DivineLife SaiBaba Breakaway TOTAL

RQ2: Improvement Application CrossCheck CrossTWebDiff Rep.Conf.TraceScreenRep.Conf. Rep.Conf. Restaurant Organizer GrantaBooks DesignTrust DivineLife SaiBaba Breakaway TOTAL

Empirical Evaluation ApplicationCrossCheckCrossTWebDiff Rep.Conf.TraceScreenRep.Conf. Rep.Conf. Restaurant Organizer GrantaBooks DesignTrust DivineLife SaiBaba Breakaway TOTAL CrossCheck detected more differences than WebDiff (62% more) and CrossT (84% more) CrossCheck reported fewer false positives than WebDiff (15% less) and CrossT (34% less)

Future Work Improved computer vision algorithms to reduce false positives and diminish noise sources 36 Perform user studies for feedback from real web developers to further improve our technique Study behavioral equivalence across different platforms, especially mobile

Related Work Industrial tools BrowserShots, Adobe Browser Lab, MS Expression Web Test Suites for Web Browsers – Acid and test262 Eaton & Memon [IJWET’07] Precursors to CrossCheck WebDiff: Roy Choudhary, Versee and Orso [ICSM’10] CrossT: Mesbah and Prasad [ICSE’11] 37

Contributions of CrossCheck 38 Machine-learning based automated detection Detects both visual and trace-level XBIs Empirical evaluation shows effectiveness Cluster CBDs into meaningful XBIs