DB-5: Simulating User Load Tom Bascom VP Technology White Star Software

DB-5: Simulating User Load - 2 Sanity Check! n This Presentation is NOT about simulating your user interface. n We are going to discuss a method for creating a database load that mimics a target usage profile.

DB-5: Simulating User Load - 3 The Challenge n A Volatile Business Environment n System Load expands to consume all resources… n Something is Always changing… n Fighting Misconceptions: –Load testing is too difficult. –Load simulation tools are expensive. –Load testing tools don’t support Progress.

DB-5: Simulating User Load - 4 Why Simulate User Load? n Regression Testing n Finding the Limits / Capacity Planning n Root Cause Analysis n Minimizing Cost / Optimizing Resources n Quality of Service (QoS) Assurances n To support other testing efforts more realistically

DB-5: Simulating User Load - 5 Considerations of Fidelity n Assumptions n Characterizing the Load n Where are the Bottlenecks? n Users are fickle… n How do we (realistically) drive test scenarios? (Or: Are we really simulating user load?)

DB-5: Simulating User Load - 6 Assumptions n We are primarily interested in the DATABASE. n READ activity is, by far, the largest consumer of database resources. n WRITE activity is also important. n An ABL session is an ABL session… (in terms of resource use) n The functionality of the application is not being tested (by these tests).

DB-5: Simulating User Load - 7 Characterizing the Load n Historical data collection is a must. n Application Metrics –Orders Taken –Widgets Produced n Baseline db Performance –Logical IO ops/sec –Disk IO ops/sec –Latch & Resource Waits –TRX, Record Locks & Blocked Clients n IO by User n TableStats

DB-5: Simulating User Load - 8 Where are the Bottlenecks? n Database –Latch contention –Transaction throughput n Operating System –Disk –CPU –Memory –Network

DB-5: Simulating User Load - 9 Users are fickle… n They work on different things… n at different rates … n pretty much randomly … n and sometimes at their leisure ;-) n Our best friends are: –IO by User (PROMON, ProTop) –_TableStat (VST, ProTop)

DB-5: Simulating User Load - 10 IO Requests by User

DB-5: Simulating User Load - 11 IO Requests by Table

DB-5: Simulating User Load - 12 IO Requests by Table Create Read Update Delete 5,940,439 1,662,688,907 2,540,165 2,500, % 99.34% 0.15% 0.15% 83.8/s 23,454.5/s 35.8/s 35.3/s Table Name Created % Read % Updated % Deleted % CHART_ENTRY ,061,326, BUREAU_QUEUE ,087, , APP_INFO 4, ,913, , BTE_QUEUE ,612, , SCORE_EXPWGHT ,349, APP_REQUEST 4, ,491, , APP_TRADE 105, ,499, , , APP_OPTION 23, ,341, , , APP_TRANSTAGG 3,347, ,603, ,919,

DB-5: Simulating User Load - 13 Data Distribution

DB-5: Simulating User Load - 14 Idle Users? Current Users: 192 Idle Users: 0: :02 8 0: :04 4 0:05 4 0:06 2 0:07 4 0:08 4 0: m - 1hr: 51 Hour+ old: 32

DB-5: Simulating User Load - 15 How do we drive test scenarios? n With some ABL code that reads and writes records as the application does. n Distributed among many sessions. n Profiled according to TableStats.

DB-5: Simulating User Load - 16 Code !!!

DB-5: Simulating User Load - 17 WARNING n Using the CREATE, UPDATE or DELETE routines will modify your database! n Use these destructive options only with test copies of your database. n Use the READ-ONLY option with production databases!

DB-5: Simulating User Load - 18 A Simulation Controller n You already have everything that you need… n A generic Load driver n Tuning the driver to your Load n Controlling test runs

DB-5: Simulating User Load - 19 Control Program (pacectl.p) cmd = “mbpro &1 -pf pace.pf -param &2 -U &3". input from value( "io.usr" ). do while true: import io_usr nm_usr. xcmd = substitute( cmd, pdbname(1), io_usr, nm_usr ). message xcmd. if opsys = "unix" then os-command silent value( xcmd ). else run spawn( input xcmd, input "", output pid ). pause 1. end.

DB-5: Simulating User Load - 20 Config File (io.usr) 0,500,50,0 jami 0,100,10,0 tucker 0,50,5,0 jami 0,40,4,0 julia 0,30,3,0 peter 0,20,2,0 emily

DB-5: Simulating User Load - 21 pace.p do while true: x = random( 1, pickLimit ). find first tt_pace where tblPick >= x. run xread( tblName, r_rate ). t = time - s. do while (( i / t ) > r ): pause 1 no-message. t = time - s. end.

DB-5: Simulating User Load - 22 xread() create buffer bh for table tblName. qh:set-buffers( bh ). qh:query-prepare( “for each “ + tblName ). j = random( j / 2, j * 10 ). do while k < j: qh:get-next( no-lock ) no-error. if qh:query-off-end then do: qh:get-first( no-lock ) no-error. next. end. k = k + 1. end.

DB-5: Simulating User Load - 23 Demo !!!

DB-5: Simulating User Load - 24 Selected Case Studies n Benchmarking OpenEdge 10.0 –PSDN Whitepaper: n Ensuring Disk Array Performance n Verifying a Failover Plan n Validating Parameter Changes n Providing a background load for development or testing environments. n Supporting Demos ;-)

DB-5: Simulating User Load - 25 Benchmarking OE10 n PSDN Whitepaper: n Problem: –How Does OE10 Compare to Progress 9?

DB-5: Simulating User Load - 26 Ensuring Disk Array Performance n Problem: –Will a new disk array configuration support the production workload? n Several previous attempts to go live had failed. n Load simulation was used to prevent a repeat of that fiasco. n Problems were found and solutions were verified and confidence was repaired prior to a successful implementation.

DB-5: Simulating User Load - 27 Verifying a Failover Plan n Problem: –Redundant (and expensive) systems were purchased and configured for mutual failover. n A method to realistically test a failover with significant load on the hardware was needed. n The failover worked very well! n But the “fail back” failed miserably  n As a result a serious OS bug was identified and fixed and downtime avoided.

DB-5: Simulating User Load - 28 Validating Parameter Changes n Problem: –Changing many Progress and OS parameters can result in unanticipated limits being reached. n # of file handles, semaphores, memory size etc… n Solution: –Use the load simulator to start sessions and stress these limits.

DB-5: Simulating User Load - 29 Background Load n Problem: –Developers and testers are often working on over-powered servers with no competition for resources. –Which leads to a false sense of good performance when testing. n Solution: –Run load simulator to provide a continuous background load.

DB-5: Simulating User Load - 30 Demos! n Problem: –Single user demos are boring. n Solution: –Create some fake users that do interesting things.

DB-5: Simulating User Load - 31 Summary n Basic Load Simulation is easier than you think! n And doesn’t necessarily require fancy (and expensive) tools. n It does require fore-thought and planning. n And can be immediately useful to you!

DB-5: Simulating User Load - 32 Questions