1. CrossCheck: Combining Crawling and Differencing to Better Detect Cross-browser Incompatibilities in Web Applications Shauvik Roy Choudhary, Mukul Prasad, Alex Orso Georgia Institute of Technology Fujitsu Labs of America

2. Move to the Web 2

3. Multitude of Browsing Environments 3

4. Georgia Tech Website Mozilla Firefox Internet Explorer 4

5. IEEE TSE Website 5

6. Granta Books Website 6 Internet Explorer Mozilla Firefox

7. Mimic end user’s perception Produce human Consumable reports Manual inspection is expensive Need automated classification ? Modern apps have many dynamic screens DOM differs between browsers 7 Ignore variable elements on webpage 15 Sep 2010 3:15pm Work with browser security controls

8. Running Example 8

9. 9

10. 10

11. Running Example 11

12. Running Example 12

13. Running Example 13

14. Running Example 14 Missing screen transitions 1

15. Running Example 15 Missing shadow 2 Wrong count of items 3

16. Running Example 16

17. Running Example 17

18. A Tale of Two Techniques 18 WebDiff Screen-level differences (graph matching, computer vision) CrossT Trace-level differences (graph isomorphism)

19. Goal: To better detect both functional and visual XBIs by combining the two complementary techniques High level view of the approach Web Application CrossCheck Cross-browser Error Report 1. Model Generation 2. Model Comparison 3. Report Generation 19

20. Screen Transition Graph 1. Model Generation Web Application DOM Tree Screen image + geometries Ajax Crawler (Crawljax) Screen Model Screen image + geometries DOM Tree 20

21. 2a. Trace Level Comparison STG Match Screen Transition Graphs from two browsers using graph isomorphism (both screens and transitions) Output: Pairs of matching screens  Perform Screen-level comparison Unmatched Screens  Report Trace-level issue Unmatched Transitions  Report Trace-level issue STG 21

22. 2a. Trace Level Comparison S0S0 S1S1 S2S2 S1’S1’S2’S2’ 22 S0’S0’

23. 2b. Screen Level Comparison Compare DOMs to find matching elements Use MatchIndex f (xPath, coords, zindex, visibility etc. ) Output: Pairs of matched DOM nodes (corr. screen elements)  for visual comparison Unmatched DOM nodes if any Given a pair of matched screens (from prev. step) 23 Match DOMs Corr. Screen elements

24. 2b. Screen Level Comparison Like-colored (matched) screen-elements compared through visual comparison 24

25. Machine Learning based Classifier Machine Learning based Classifier YES NO Visual Comparison Extract Features Features Used: Size difference Position difference/ displacement Absolute size Text value difference (based on DOM)  2 –Image distance (between images of 2 elements)  ? Do these screen elements look the same ? 25

26. Machine Learning based Classifier Training the classifier (offline – single time) Feature Extraction False True Screens with (known) Cross-Browser Differences Set of screen-element comparison instances 26

27. 3. Report Generation (Clustering) 27

28. Empirical Evaluation RQ1 (Effectiveness) Can CrossCheck identify different kinds of CBDs in real-world web applications and correlate them to identify XBIs? RQ2 (Improvement) How effective is CrossCheck when compared to CrossT and WebDiff? 28

29. Subjects: Experimental Setup Web AppURLType STDOM Nodes minmaxavg Restaurant http://localhost/restaurant Information38785846821 Organizer http://localhost/organizer Productivity1399100012748213051 GrantaBooks http://grantabooks.com Publisher98156253780025852 DesignTrust http://designtrust.org Business102077722643718694 DivineLife http://sivanandaonline.org Spiritual109908214061149886 SaiBaba http://shrisaibabasansthan.org Religious13205244260612162 Breakaway http://breakaway-adventures.com Sport91881914514813059 29

30. Procedure: 1.Used latest versions of Firefox (v7.0.1) and Internet Explorer (v9.0.3) 2.Ran CrossCheck on the subjects to perform the various phases of the technique 3.Manually checked results for false positives and negatives Experimental Setup 30

31. RQ1: Effectiveness ApplicationTracePositionSizeVisibilityTextAppearanceCBDsXBIs Restaurant 402023119 Organizer 140425016218 GrantaBooks 160011012816 DesignTrust 423920146189130 DivineLife 70031708173 SaiBaba 2531735510389 Breakaway 01313200246391268 TOTAL 4720246286522865603 31

32. RQ1: Effectiveness ApplicationTracePositionSizeVisibilityTextAppearanceCBDsXBIs Restaurant 402023119 Organizer 140425016218 GrantaBooks 160011012816 DesignTrust 423920146189130 DivineLife 70031708173 SaiBaba 2531735510389 Breakaway 01313200246391268 TOTAL 4720246286522865603 32

33. RQ2: Improvement Application CrossCheck CrossTWebDiff Rep.Conf.TraceScreenRep.Conf. Rep.Conf. Restaurant 11 47 6 5 Organizer 6250143620214288 GrantaBooks 2827161134816109 DesignTrust 189274236809821 DivineLife 811376174110678 SaiBaba 103362341883425 Breakaway 3911500 306029163 TOTAL 86531447267286449547119 33

34. RQ2: Improvement Application CrossCheck CrossTWebDiff Rep.Conf.TraceScreenRep.Conf. Rep.Conf. Restaurant 11 47 6 5 Organizer 6250143620214288 GrantaBooks 2827161134816109 DesignTrust 189274236809821 DivineLife 811376174110678 SaiBaba 103362341883425 Breakaway 3911500 306029163 TOTAL 86531447267286449547119 34

35. Empirical Evaluation ApplicationCrossCheckCrossTWebDiff Rep.Conf.TraceScreenRep.Conf. Rep.Conf. Restaurant 11 47 6 5 Organizer 6250143620214288 GrantaBooks 2827161134816109 DesignTrust 189274236809821 DivineLife 811376174110678 SaiBaba 103362341883425 Breakaway 3911500 306029163 TOTAL 86531447267286449547119 35 CrossCheck detected more differences than WebDiff (62% more) and CrossT (84% more) CrossCheck reported fewer false positives than WebDiff (15% less) and CrossT (34% less)

36. Future Work Improved computer vision algorithms to reduce false positives and diminish noise sources 36 Perform user studies for feedback from real web developers to further improve our technique Study behavioral equivalence across different platforms, especially mobile

37. Related Work Industrial tools BrowserShots, Adobe Browser Lab, MS Expression Web Test Suites for Web Browsers – Acid and test262 Eaton & Memon [IJWET’07] Precursors to CrossCheck WebDiff: Roy Choudhary, Versee and Orso [ICSM’10] CrossT: Mesbah and Prasad [ICSE’11] 37

38. Contributions of CrossCheck 38 Machine-learning based automated detection Detects both visual and trace-level XBIs Empirical evaluation shows effectiveness Cluster CBDs into meaningful XBIs