Deriving Test Data for Web Applications from User Session Logs Jeff Offutt SWE 737 Advanced Software Testing

User-Session Testing User session : A collection of user requests in the form of URL and name-value pairs Stored in logs on the server The name-value pairs are stored in logs on the server User sessions to tests : Each logged request is turned into an HTTP request Automated with a tool like HTTPUnit or Selenium SWE 737 © Jeff Offutt

User-Session Testing Example An efficient way to generate tests Empirical studies have found success in finding faults SWE 737 © Jeff Offutt

Issues and Problems Hard to scale with large numbers of user sessions Tends to create “happy path tests,” not exceptional or unusual cases SWE 737 © Jeff Offutt

1. Prioritizing Large Numbers of Tests Researchers have tried to put tests in priority order according to various criteria Semantic differences between programs Coverage of test requirements Fault exposure potential Likelihood of a fault occurring Length (or size) of tests SWE 737 © Jeff Offutt

Criteria by Sampath et al. Length-Based Base Request Long to Short (Req-LtoS) Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL) Frequency-Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS) Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way) SWE 737 © Jeff Offutt

Length-Based Prioritization Order by the number of HTTP base requests in a test case Length of a test case is the number of base requests the test case contains (counting duplicates) Descending order of length (Req-LtoS) Ascending order of length (Req-StoL) SWE 737 © Jeff Offutt

Length-Based Prioritization Order tests by the number of parameters PV-LtoS: Longest parameter list to shortest PV-StoL: Shortest parameter list to longest SWE 737 © Jeff Offutt

Frequency-Based Prioritization Order the pages (JSPs and Servlets) in terms of how many times they are accessed Prioritize tests that cover pages that are more frequently accessed (1-way) Also order pages that are used in sequences of length 2 (pairs, 2-way) SWE 737 © Jeff Offutt

Parameter Value Coverage Prioritization Unique Coverage of Parameter Values (1-way) Select the next test that maximizes the number of parameter-value pairs that have NOT been used Example (blue is new PV pairs): SWE 737 © Jeff Offutt

Parameter Value Coverage Prioritization Parameter-Value Interaction Coverage (2-way) Select the next test that maximizes the number of 2-way parameter-value interactions between pages that occur in some test SWE 737 © Jeff Offutt

Experimental Evaluation (Sampath 2008) Basis for evaluation: Fault detection rate Average percent of faults detected Execution time of the test suite Subject applications: Book—e-commerce bookstore CPM—create grader accounts for TA Masplas—web app for regional workshop SWE 737 © Jeff Offutt

Evaluation Metrics Rate of fault detection Finding the most faults in the first 10% of tests executed Percent tests needed to find all faults Average percentage of faults detected By all tests SWE 737 © Jeff Offutt

Types of Faults Seeded Data Store—faults that exercise code interacting with data Logical—code logic errors in control flow Form-based Appearance—change in perceiving user’s view of the page Links—change of hyperlink locations SWE 737 © Jeff Offutt

Results Summary Book CPM Masplas Finding the most faults in 10% of tests 1-way parameter value interaction All accessed sequences (AAS) Base request long to short (Req-LtoS) Find all the faults with fewest tests 1-way and most frequently accessed sequence (MFAS) 2-way parameter value interaction SWE 737 © Jeff Offutt

Conclusions Study did not determine which is best. Probably depends on the app. Almost everything was better than random SWE 737 © Jeff Offutt

2. Testing Exceptional and Unusual Situations Literature identified two problems (reference [1], Sampath 2008) Scalability Tests focus on “happy paths” I couldn’t find any publications that address the happy path problem SWE 737 © Jeff Offutt

References Prioritizing User-session-based Test Cases for Web Applications Testing, Sreedevi Sampath, Renée Bryce, Gokulanand Viswanath, Vani Kandimalla, and Güneş Koru, First International Conference on Software Testing, Verification, and Validation, April 2008 Automated Replay and Failure Detection for Web Applications, Sara Sprenkle, Emily Gibson, Sreedevi Sampath, and Lori Pollock, International Conference of Automated Software Engineering, November 2005 Web Application Testing with Customized Test Requirements—An Experimental Comparison Study, Sreedevi Sampath, Sara Sprenkle, Emily Gibson, and Lori Pollock, International Symposium on Software Reliability Engineering, November 2006 Applying Concept Analysis to User-Session-Based Testing of Web Applications, Sreedevi Sampath, Sara Sprenkle, Emily Gibson, Lori Pollock, and A. S. Greenwald, IEEE Transactions on Software Engineering, 33(10):643-658, October 2007 SWE 737 © Jeff Offutt