Using Mutation Analysis to Test Web Applications Jeff Offutt SWE 737 Advanced Software Testing Primary source: Upsorn Praphamontripong’s PhD Dissertation
Web mutation operators Summary Process Use mutation analysis to improve our ability to test web applications Can web mutation testing reveal web faults that other testing criteria do not detect? Revised and extended Web mutation operators A Web fault model Evaluate operators SWE 737 © Jeff Offutt
Challenges in Testing Web App User’s ability to control web apps Identifying web resources with URLs Communication depending on HTTP requests Communication via data exchanges Novel control connections Server-side state management Client-side state management Let’s look at three … SWE 737 © Jeff Offutt
Challenge #1–User’s Ability to Control Web Apps Potential faults: Unintended transitions index login browse record_add submit a form post (userid , password) redirect record_insert post (name, category, content) back component simple link transition form link transition redirect transition operational transition SWE 737 © Jeff Offutt
Challenge #5–Novel Control Connections Potential faults: Inappropriate use of redirect or forward transitions submit a form post (userid , password) index login browse record_add redirect record_insert post (name, category, content) back component simple link transition form link transition redirect transition operational transition forward SWE 737 © Jeff Offutt
Challenge #6–Server-side State Management Potential faults: Inappropriate scope set Inappropriate session object initialization Omitting necessary session info application session request forward page request Client 1 request request forward page request Client 2 page SWE 737 © Jeff Offutt
Web Mutation Operators (by challenges) Users’ ability to control web apps FOB WHID WHIR WOID Identifying web resources w/ URLs WLUR WLUD WFUR WRUR WCUR Comm. via HTTP requests Comm. via data exchanges WSCR WSIR WSAD Novel control connection Server-side state management WCTR WFTR WPVD Client-side state management SWE 737 © Jeff Offutt
Mutation Operator for Faults due to Operational Transitions (challenge 1) FOB – FailOnBack <html> … <body> <body onload=“manipulatehistory()”> <script src=“failOnBack.js”></script> </html> failOnBack.js function manipulatehistory() { var currentpage = window.document.toString(); var currenturl = window.location.href; var pageData = window.document.toString(); // add a dummy url right before the current url history.replaceState ( pageData, “dummyurl”, “failonback.html” ); history.pushState ( currentpage, “currenturl”, currenturl); } // update the page content window.addEventListener ( ‘popstate’, function (event) { window.location.reload(); }); SWE 737 © Jeff Offutt
Mutation Operator for Faults due to Novel Control Connections (challenge 5) WCTR – Control transition replacement public class logout extends HttpServlet { … public void doGet( ... ) ... response.sendRedirect (URL1); getServletContext().getRequestDispatcher (URL1) .forward (request, response); } SWE 737 © Jeff Offutt
Mutation Operators for Faults due to Server-Side State Management (challenge 6) WSCR – Scope replacement <html> … <jsp:useBean id = id1 scope = “session” class = class1 /> <jsp:useBean id = id1 scope = “request” class = class1 /> </html> WSIR – Session initialization replacement public class logout extends HttpServlet { public void doGet( ... ) session = request.getSession (true); session = request.getSession (false); ... } } WSAD – Session setAttribute deletion session.setAttribute (attr1 , value1 ); // session.setAttribute (attr1 , value1 ); SWE 737 © Jeff Offutt
Mutation Operators for Faults due to Client-Side State Management (challenge 7) WHID – Hidden form input deletion <html> … <form action = URL1 method = “POST”> <input type = “hidden” name = name1 value = value1 > <!-- input type = “hidden” name = name1 value = value1 --> </html> WHIR – Hidden form input replacement <input type = “hidden” name = name1 value = value2 > WOID – Read-only input deletion <input type = “text” readonly name = name1 … > <!-- input type = “text” readonly name = name1 … --> SWE 737 © Jeff Offutt
Run tests on subject program Empirical Evaluation 11 Subject programs Run tests on subject program no Distinguishable result? yes Record killed mutants Apply mutation operators Generate tests (12 testers) mutants RQ1: How well do traditional tests kill web mutants? RQ2: Can hand designed tests kill web mutants? 11 subjects (mutate source code -- JSPs and Java servlets) 12 test teams – manually create tests using input space, logic, functional requirement, random (each test team created 1 test set for each subject each test team created 11 test sets) Experimental setup Generate web mutants (use all 15 ops 15 kinds of web mutants) Create web mutation tests execute on mutants record # mutants killed of each type Create traditional tests execute on mutants record # mutants killed of each type Compare # killed by web mutation tests vs traditional tests Compute usefulness = # unkilled of mutants / # mutants generated for each type of mutants The more mutants not kill, the higher the usefulness op gen this kind of mutants is good, use it! Run tests on mutants 2 groups of tests: web mutation tests and traditional tests SWE 737 © Jeff Offutt
Mutants and Web Mutation Tests Subjects Components LOC Mutants Equivalent Killed #Tests BSVoting (S1) 11 930 27 26 check24online (S2) 1 1619 8 5 computeGPA (S3) 2 581 18 14 conversion (S4) 388 3 faultSeeding (S5) 1541 115 16 99 34 HLVoting (S6) 12 939 103 102 31 KSVoting (S7) 7 1024 17 quotes (S8) 537 randomString (S9) 285 StudInfoSys (S10) 1766 9 6 webstutter (S11) 126 4 Total 51 9709 337 320 149 Limitation We limited the number of subjects due to manual effort (reviewing and cleaning up tests to make sure they can be fully automated). We use J2EE web app because of the availabilty 17 equivalent mutants – all are WSAD mutants - 1 in S6, attr is never accessed - 16 in S5, attrs are set before they are reset (not even accessed yet) – unnecessary set SWE 737 © Jeff Offutt
Mutants and Traditional Tests FOB WCTR WCUR WFTR WFUR WHID WHIR WLUD WLUR WOID WPVD WRUR WSAD WSCR WSIR Total #mutants killed / test Mutants generated 18 26 3 36 35 7 20 2 13 117 12 320 Testers #Tests Mutants killed T1 86 17 22 27 5 1 9 76 4 173 2.01 T2 78 21 25 75 160 2.05 T3 168 16 6 10 59 165 0.98 T4 124 29 63 174 1.40 T5 102 11 19 23 8 57 148 1.45 T6 94 15 24 53 139 1.48 T7 118 28 51 154 1.31 T8 120 56 135 1.13 T9 93 30 60 175 1.88 T10 85 46 1.41 T11 95 14 169 1.78 T12 77 128 1.66 Average 103 153 1.49 Total LOC, and mutants generated for all subjects Biggest test set (T3) Strongest test set (T4) Weakest test set (T2) Though average #tests of traditional tests is lower than #tests of mutation testst (103 vs 149), traditional tests killed on average not even half of web mutants while mutation tests killed all web mutants # mutants vary due to the features in the subjects .. Some subjects lack of certain features (e.g., hidden input and jsp:param), thus some kinds of mutants were not generated Though the total #mutants killed by some test sets may be lower, #tests can be a factor thus, consider #mutants killed / test Mu-tests #mutans killed /test = 320/149 = 2.15 None of traditional tests kill FOB, WOID, and WPVD The kinds of mutants killed the most – WFUR and WHID SWE 737 © Jeff Offutt
Mutation Scores of Traditional Tests The biggest test set (T3) -- score 0.48 The strongest test set (T4) – score 0.52 The lowest test set (T12) -- score 0.42 Average score 0.47 Web mutation tests – score 1.00 SWE 737 © Jeff Offutt
Usefulness of Web Mut Operators Why do we need this table? Remind audience that our experiment is to evaluate effectiveness of web mutation operators in terms of the kinds of web mutants other test criteria have difficulty killing RQ1: What kinds of web mutants traditional tests cannot kill? how well traditional tests do on web mutants RQ2: Can hand designed tests kill all web mutants? yes, they can be killed. If testers use our criteria, they will find more / different kinds of faults – cover faults they overlooked For WOID and WPVD, though the average usefulness is high, we can’t confidentially conclude because the number of mutants generated of these kinds are low. need more experiment to confirm SWE 737 © Jeff Offutt
Summary Analysis On average, traditional tests killed only 47% of web mutants None of traditional tests killed any FOB (failOnBack) mutants Most traditional tests missed WSCR, WLUD, and WLUR mutants (URL changes) Only 5% of web mutants were equivalent (all were session attribute deletion, WSAD) SWE 737 © Jeff Offutt
Run tests on subject program Distinguishable result? Conclusions Evaluated operators 11 web apps, 12 test teams mutants Subject program Apply mutation operators Run tests on subject program Run tests on mutants Generate tests Distinguishable result? no Record killed yes 15 web mutation operators Web fault model Future work: Conduct a fault study Evaluate web mutation tests on traditional mutants Modify the tool to use program schema Support Ajax and JavaScript SWE 737 © Jeff Offutt
References Testing Web Applications with Mutation Analysis, Upsorn Praphamontripong, PhD Dissertation, George Mason University, 2017 (expected) Finding Redundancy in Web Mutation Operators, Upsorn Praphamontripong and Jeff Offutt, Thirteenth IEEE Workshop on Mutation Analysis, April 2017, Tokyo, Japan An Experimental Evaluation of Web Mutation Operators, Upsorn Praphamontripong, Jeff Offutt, Lin Deng, and JingJing Gu, Twelfth IEEE Workshop on Mutation Analysis, April 2016, Chicago Illinois, USA Applying Mutation Testing to Web Applications, Upsorn Praphamontripong and Jeff Offutt. Sixth Workshop on Mutation Analysis, April 2010, Paris, France SWE 737 © Jeff Offutt