Prioritizing User-session-based Test Cases for Web Applications Testing Sreedevi Sampath, Renne C. Bryce, Gokulanand Viswanath, Vani Kandimalla, A.Gunes Koru University of Maryland University of Nevada Sreedevi Sampath, Renne C. Bryce, Gokulanand Viswanath, Vani Kandimalla, A.Gunes Koru University of Maryland University of Nevada

Demand in Secure & Reliable Web Applications A web application is a set of static or dynamic web pages that are accessible through a browser over a network  Critical part of business for many organizations  Must be available 24/7  Failures in this domain result in losses of millions of dollars A web application is a set of static or dynamic web pages that are accessible through a browser over a network  Critical part of business for many organizations  Must be available 24/7  Failures in this domain result in losses of millions of dollars

Complications for Automated Testing  Fix bugs and deploy a new version within a short time frame  Number of integration technologies - third party reusable modules - a well-defined, layered architecture - dynamically generated pages with dynamic context - extensions to an application framework  Changes in database information  Fix bugs and deploy a new version within a short time frame  Number of integration technologies - third party reusable modules - a well-defined, layered architecture - dynamically generated pages with dynamic context - extensions to an application framework  Changes in database information

Previous Work: Testing with User-Session-based Test  Logs of actual usage data are converted into test cases to run regression tests (Spenkle et al, 2005)  Efficient at detecting faults but unscalable with larger numbers of user sessions (Elbaum et al. 2005)  Reduced user-session-based test suites on “covering all base requests” criterion (Sampath 2007)  Logs of actual usage data are converted into test cases to run regression tests (Spenkle et al, 2005)  Efficient at detecting faults but unscalable with larger numbers of user sessions (Elbaum et al. 2005)  Reduced user-session-based test suites on “covering all base requests” criterion (Sampath 2007)

Previous Work: Prioritizing Test Cases Schedule the test cases according to some criterion to satisfy a performance goal Criteria: semantic differences between 2 programs (Binkley 1992) coverage of requirements (Offut 1995) fault exposure potential (Rothermel 2001) fault likelihood (Elbaum 2002) length of tests for GUI-based programs (Bryce & Memon 2007) All strategies prioritize test cases for C and Java programs. Schedule the test cases according to some criterion to satisfy a performance goal Criteria: semantic differences between 2 programs (Binkley 1992) coverage of requirements (Offut 1995) fault exposure potential (Rothermel 2001) fault likelihood (Elbaum 2002) length of tests for GUI-based programs (Bryce & Memon 2007) All strategies prioritize test cases for C and Java programs.

Prioritization for Web Application Testing This work  Expands upon previous prioritization work to develop strategy specifically for web application testing  Empirically evaluates strategies using user-session-based test suites  Gives guidance to testers based on the results of their empirical evaluation This work  Expands upon previous prioritization work to develop strategy specifically for web application testing  Empirically evaluates strategies using user-session-based test suites  Gives guidance to testers based on the results of their empirical evaluation

Motivation and Approach Outline Criterion:  Length Based  Base Request Long to Short (Req-LtoS)  Base Request Short to Long (Req-StoL)  Parameter Value Long to Short (PV-LtoS)  Parameter Value Short to Long (PV-StoL)  Frequency Based  Most Frequently Accessed Sequence (MFAS)  All Accessed Sequences (AAS)  Parameter Value Coverage  Unique Coverage of Parameter Values (1-way)  2-way Parameter-Value Interaction Coverage (2-way) Criterion:  Length Based  Base Request Long to Short (Req-LtoS)  Base Request Short to Long (Req-StoL)  Parameter Value Long to Short (PV-LtoS)  Parameter Value Short to Long (PV-StoL)  Frequency Based  Most Frequently Accessed Sequence (MFAS)  All Accessed Sequences (AAS)  Parameter Value Coverage  Unique Coverage of Parameter Values (1-way)  2-way Parameter-Value Interaction Coverage (2-way) Test suite prioritization uses the entire test suite for execution but the test cases are ordered based on criteria that attempts to detect faults as quickly as possible in test execution cycle

Approach: Test Case Prioritization Criteria: Test Length Order by the number of HTTP base requests in a test case  Descending order of length (Req-LtoS)  Ascending order of length (Req-StoL), where length of a test case is defined as the number of base requests the test case contains counting duplicates. Order by the number of HTTP base requests in a test case  Descending order of length (Req-LtoS)  Ascending order of length (Req-StoL), where length of a test case is defined as the number of base requests the test case contains counting duplicates.

Approach: Test Case Prioritization Criteria: Frequency-based Prioritization Test cases that cover most frequently accessed pages/sequence of pages are selected for execution before test cases that exercise the less frequently accessed pages/sequences of pages in the entire test suite Sequences are considered:  in terms of base requests  that involve interactions between JSP and JSP servlet pages  of size 2 Test cases that cover most frequently accessed pages/sequence of pages are selected for execution before test cases that exercise the less frequently accessed pages/sequences of pages in the entire test suite Sequences are considered:  in terms of base requests  that involve interactions between JSP and JSP servlet pages  of size 2

Two Ways to Prioritize Frequently Accessed Sequences  All Accessed Sequences (AAS) For each sequence in the application, beginning with the most frequently accessed sequence, test cases that have max. occurrences of these sequences are selected for execution before other test cases in the test suite  All Accessed Sequences (AAS) For each sequence in the application, beginning with the most frequently accessed sequence, test cases that have max. occurrences of these sequences are selected for execution before other test cases in the test suite  Most Frequently Accessed Sequence (MFAS) Orders test cases in decreasing order of the number of times that sequence appears in the test case

Systematic Prioritization by Parameter-Values Pages that contain parameters for which users may specify values Criterion:  Length Based  Base Request Long to Short (Req-LtoS)  Base Request Short to Long (Req-StoL)  Parameter Value Long to Short (PV-LtoS)  Parameter Value Short to Long (PV-StoL)  Frequency Based  Most Frequently Accessed Sequence (MFAS)  All Accessed Sequences (AAS)  Parameter Value Coverage  Unique Coverage of Parameter Values (1-way)  2-way Parameter-Value Interaction Coverage (2-way)

Systematic Prioritization by Parameter-Values Unique Parameter-Value Coverage The 1-way criterion selects a next test that maximizes the number of parameter-values that have not appeared in previously selected tests

Systematic Prioritization by Parameter-Values Parameter-Value Interaction Coverage The 2-way criterion selects a next test that maximizes the number of 2-way parameter-value interactions between pages that occur in a test The 2-way criterion selects a next test that maximizes the number of 2-way parameter-value interactions between pages that occur in a test

Systematic Prioritization by Parameter-Values Length by Parameter-Value Counts  Prioritize test by the number of parameter- values in a test case

Approach Outline Criterion:  Length Based  Base Request Long to Short (Req-LtoS)  Base Request Short to Long (Req-StoL)  Parameter Value Long to Short (PV-LtoS)  Parameter Value Short to Long (PV-StoL)  Frequency Based  Most Frequently Accessed Sequence (MFAS)  All Accessed Sequences (AAS)  Parameter Value Coverage  Unique Coverage of Parameter Values (1-way)  2-way Parameter-Value Interaction Coverage (2-way) Criterion:  Length Based  Base Request Long to Short (Req-LtoS)  Base Request Short to Long (Req-StoL)  Parameter Value Long to Short (PV-LtoS)  Parameter Value Short to Long (PV-StoL)  Frequency Based  Most Frequently Accessed Sequence (MFAS)  All Accessed Sequences (AAS)  Parameter Value Coverage  Unique Coverage of Parameter Values (1-way)  2-way Parameter-Value Interaction Coverage (2-way)

Experimental Methodology

Experimental Evaluation Evaluation is based on:  Fault detection rate  Average percent of faults detected  Execution time of the test suite Subject Applications (test suites):  Book - e-commerce bookstore  CPM - create grader accounts for TA  Masplas – web app for regional workshop Evaluation is based on:  Fault detection rate  Average percent of faults detected  Execution time of the test suite Subject Applications (test suites):  Book - e-commerce bookstore  CPM - create grader accounts for TA  Masplas – web app for regional workshop

Evaluation Metrics  Rate of fault detection  Average percentage of faults detected  Rate of fault detection  Average percentage of faults detected Study is based on:  Finding most faults in earliest tests (first 10% of tests executed)  Finding 100% of the faults earliest

Experimental Evaluation (contd…) Categories of fault seeded:  Data Store - faults that exercise code interacting with data  Logical - code logic errors in control flow  Form based  Appearance - change in perceiving user’s view of the page  Links - change of hyperlink locations Categories of fault seeded:  Data Store - faults that exercise code interacting with data  Logical - code logic errors in control flow  Form based  Appearance - change in perceiving user’s view of the page  Links - change of hyperlink locations

Result Analysis for CPM  AAS is most effective technique to find most faults in the earliest test  2-way parameter value interaction is most effective technique to find all the faults at the earliest

Result Analysis for Masplas  Req-LtoS - Test length based on number of base requests (descending) is most effective technique to find most faults in the earliest test  2-way parameter value interaction is most effective technique to find all the faults at the earliest

Result Analysis for Book  1-way parameter value interaction is most effective technique to find most faults in the earliest test  MFAS and 1-way parameter value interaction are most effective techniques to find all the faults at the earliest

Summary of results  Choice of the testing technique is based on the tester’s goal and characteristics of web application  The 2-way parameter value interaction and PV-LToS technique is to be used when the primary goal is to find 100% of faults at the earliest  Frequency based technique needs to be used to validate the functionality of frequently used web pages of an application  Choice of the correct prioritization would help in finding and fixing the faults in application quickly, thus saving cost impact  Choice of the testing technique is based on the tester’s goal and characteristics of web application  The 2-way parameter value interaction and PV-LToS technique is to be used when the primary goal is to find 100% of faults at the earliest  Frequency based technique needs to be used to validate the functionality of frequently used web pages of an application  Choice of the correct prioritization would help in finding and fixing the faults in application quickly, thus saving cost impact

Future Work  Consideration of costs associated with prioritization strategies  Evaluate hybrid approaches to prioritization (combining prioritization techniques)  Extension to frequency based metrics to include other metrics  Consideration of costs associated with prioritization strategies  Evaluate hybrid approaches to prioritization (combining prioritization techniques)  Extension to frequency based metrics to include other metrics

Conclusion Prioritization criteria improve the rate of fault detection of test suites when compared to random ordering of test cases