How to Design and Implement Load Testing for Sakai The Michigan Experience

8 Dec 2 Why Load Test? –Measure end-user response time How long does a specified operation take? Currently a minor part of our testing –Define optimal hardware configuration Enables pre-tuning of Oracle Testing various JVM settings –Check reliability How long or hard can system work without producing errors (failures)?

8 Dec 3 Why Load Test.... –Check hardware & software upgrades Does new host or software perform as well as previously tested versions? –Evaluate new products Used to evaluate hardware for database replacement –Measure system capacity How much can the system handle before performance degradation occurs? –Identify bottlenecks What part of the environment causes performance degradation?

8 Dec 4 Test Environment Performance Testing Tool Infrastructure Courses & Users Data

8 Dec 5 Performance Testing Tool Mercury Interactive LoadRunner –VuGen (virtual user scripting) –Controller (scenario and load generation) –Analyzer (analysis and reports) Approach appropriate to any tool Presentation will not focus on tool

8 Dec 6 Infrastructure As close to Production as possible –Hardware –Software Same configuration unless testing something specific –Authentication Use application rather than CoSign ›UM-developed, open-source single web login service –UMIAC Warehouse for needed administrative data ›UM-developed, PERL scripts accessing Peoplesoft system –Database servers not always identical Where less “powerful,” larger host will likely perform better than observed in tests –AFS resource storage

8 Dec 7 Courses & Users UMIACload provides course names, instructor owners, and rosters of students –11,000 courses –11,000 “instructors” –43,000 “students” Why so many? –To permit high level of activity without excessive concurrency

8 Dec 8 Data Testing requires that all courses be pre- populated with materials to be used by students Both AFS and Oracle data used Saved versions of “base” data maintained to enable rollback to “clean” data Always start with data refresh when adding new tools to test suite

8 Dec 9 Testing Process Scripts Scenarios Analysis Results

8 Dec 10 Script Setup Analyze tool to be scripted Understand tool usage –Value of user profiling Design test plan –Goals –Single or multiple scripts? –Student or Instructor activity? –Are course materials required? Design data files –Used by Virtual Users (VU) to change input parameters

8 Dec 11 Script Setup... Create course materials Populate courses with materials –Must be done sequentially to ensure all courses get materials –Multiple resources can be added to same course with same script –Need more efficient method Record activity to be tested

8 Dec 12 Script Parameterization Scripts recorded as single user Load iterations need to change users each iteration –Data files used to manage user & input changes All object-oriented elements in Sakai receive associated ID numbers that must change whenever a course or user changes –Recording rules used to manage ID number changes –Generic parameter identifiers changed to make analysis easier

8 Dec 13 Script Pseudocode - Chat Login as “u10001” –Moves to “My Workspace” Select course tab –Use “LOADTEST ” Select “Chat” tool from left-hand navigation column –Moves to Chat tool Enter message into Chat text entry box Return to “My Workspace” Logout

Chat() { web_set_max_html_param_len("1024"); web_url("xlogin", "URL= "Resource=0", "RecContentType=text/html", "Referer=", "Snapshot=t1.inf", "Mode=HTML", EXTRARES, "Url=../ctlib/skin/ctools/images/logo_inst.gif", "Referer= ENDITEM, "Url=../ctlib/skin/ctools/images/topleft-p-title.gif", "Referer= ENDITEM, "Url=../ctlib/skin/ctools/images/help.gif", "Referer= ENDITEM, "Url=../ctlib/skin/ctools/images/topright-p-title.gif", "Referer= ENDITEM, "Url=../ctlib/skin/ctools/images/reload_h.gif", "Referer= ENDITEM, "Url=../ctlib/skin/ctools/images/chatpresicon.gif", "Referer= ENDITEM, LAST); Script Walkthrough - Chat

// [WCSPARAM WCSParam_UserID 36 b2de7f47-7c74-40d d191450a5d] Parameter {WCSParam_UserID} created by Correlation Studio web_reg_save_param( "WCSParam_UserID", "LB=site/", "RB=\"", "Ord=2", "Search=Body", "RelFrameId=1.1", LAST ); web_submit_form("xlogin_2", "Snapshot=t2.inf", ITEMDATA, "Name=eid", "Value={CTlogin}", ENDITEM, "Name=pw", "Value={CTpwd}", ENDITEM, "Name=submit", "Value=Login", ENDITEM, EXTRARES, "Url=../ctlib/skin/ctools/images/topright-tab.gif", "Referer= ENDITEM, "Url=../ctlib/skin/ctools/images/topright-curtab.gif", "Referer= ENDITEM, "Url=../ctlib/skin/ctools/images/topright-tab_hov.gif", "Referer= ENDITEM, LAST);

web_link("LOADTEST {Course2} 001 W06", "Text=LOADTEST {Course2} 001 W06", "Snapshot=t3.inf", LAST); lr_think_time(5); // [WCSPARAM WCSParam_ChatID d7683-e9fc-41d f0ac6fea0d60] Parameter {WCSParam_ChatID} created by Correlation Studio web_reg_save_param( "WCSParam_ChatID", "LB=title/", "RB=?", "Ord=1", "Search=Body", "RelFrameId=1.3", LAST ); web_link("Chat Room", "Text=Chat Room", "Snapshot=t4.inf", EXTRARES, "Url=/courier/194d7683-e9fc-41d f0ac6fea0d60/ ?auto=true", "Referer=", ENDITEM, "Url=/courier/194d7683-e9fc-41d f0ac6fea0d60/ ?auto=true", "Referer=", ENDITEM, "Url=/courier/194d7683-e9fc-41d f0ac6fea0d60/ ?auto=true", "Referer=", ENDITEM, "Url=/courier/{WCSParam_UserID}-presence/ ?auto=true", "Referer=", ENDITEM, LAST);

web_url("{WCSParam_UserID}", "URL= "Resource=0", "RecContentType=text/html", "Referer= "Snapshot=t5.inf", "Mode=HTML", EXTRARES, "Url=/courier/{WCSParam_ChatID}/ ?auto=true", "Referer=", ENDITEM, LAST);

web_submit_data("{WCSParam_ChatID}", "Action= "Method=POST", "RecContentType=text/html", "Referer= "Snapshot=t6.inf", "Mode=HTML", ITEMDATA, "Name=message", "Value=Who me? Wadya want, meathead? sheesh.....", ENDITEM, "Name=eventSubmit_doSend", "Value=x", ENDITEM, "Name=send", "Value=Add message", ENDITEM, EXTRARES, "Url=/courier/{WCSParam_ChatID}/ ?auto=true", "Referer=", ENDITEM, LAST); web_url("{WCSParam_ChatID}_2", "URL= "Resource=0", "RecContentType=text/html", "Referer= "Snapshot=t7.inf", "Mode=HTML", LAST);

lr_think_time(6); web_url("%7E{CTlogin}", "URL= "Resource=0", "RecContentType=text/html", "Referer= "Snapshot=t8.inf", "Mode=HTML", LAST); return 0; }

8 Dec 20 Scenarios One or more scripts to be run in load test –Number of virtual users running script –Number of iterations per virtual user –Pacing (how much delay between iterations) –Think time (emulates user variability) –Browser emulation –Network bandwidth emulation –Results detail (step or full transaction level)

8 Dec “Core” Tests Instructor –CreateAnnounce –CreateAssign –ReviseResource –ScheduleEvent –UploadLgFile –UploadSmFile Student –Chat –CreateProject –DownloadSmFile –DownloadLgFile –NewDiscussion –ReadBrowse –SubmitAssign

8 Dec 22 “New” Tool Tests Instructor –AddSoMod –AddVzMod –EditSyllabus –EditWiki3L –SubmitGrades Student –ReadTxtMod –ReadSoMod –ReadVzMod –ReadSyllabus –ReadWiki3L –ViewGrade

8 Dec 23 Scenario Preparation Single script runs –1 VU –100 iterations –No wait time between iterations –No think time –Showing response times for each step of transaction Identify serious script problems Take average response time for scenario

8 Dec 24 Scenario Iteration Timing (125%) Script name(VU, Wait) CreateSite/Project3VU, 0sec UploadSmFile (85%)4VU, 0sec UploadLgFile (15%)2VU, 7sec DownloadSmFile (85%)78VU, 0sec DownloadLgFile (15%)10VU, 0sec CreateAnnounce2VU, 0sec ScheduleEvent6VU, 0sec NewDiscussion1VU, 1sec CreateAssign1VU, 57sec SubmitAssign8VU, 0sec ReviseResource2VU, 4sec Chat10VU, 0sec LoginLogout & ReadBrowse76VU, 0sec Script name(VU, Wait) EditWiki3L1VU, 136sec ReadWiki3L1VU, 1sec EditSyllabus1VU, 157sec ReadSyllabus1VU, 29sec SubmitGrades2VU, 67sec ViewGrades1VU, 12sec AddVzMod1VU, 140sec AddSoMod1VU, 141sec ReadTxtMod1VU, 36sec ReadVzMod1VU, 23sec ReadSoMod1VU, 44sec

8 Dec 25 Analysis & Results Scenario runs for 1 hour plus ramp up and down time Upon completion of run, use LoadRunner Analyzer to crunch data Compare resulting graphs with previous runs Developers, DBA, SysAdmins look for performance bottlenecks Fixes proposed and test rerun

8 Dec 26 CTload Scenario Results: 2.0_r3636w8app (100%)

8 Dec 27 CTload Scenario Results: 2.2.1_r14159w6app (125%)

8 Dec 28 Sharing the Burden? Multiple campuses participate –Ideally a few dedicated partners Localized hardware Localized test tool choice Centralized resources –Test design –Pseudocode for scripts –Course data files –Results –Support and discussion

8 Dec 29 Community Testing? Share common core tests –Baseline for comparison Focus on new tools of interest to campus –Contribute resources back to central repository –Contribute results to community Greater stress on performance in development