2. DB-14: Tales of the Bunker - 2005 Gus Björklund, Progress Software Corporation John Harlow, Bravepoint, Inc. Dan Foreman, Bravepoint, Inc. Rich Banville, Progress Software Corporation

3. © 2006 Progress Software Corporation3 DB-14 Tales of the Bunker 2005 Goals of the Bunker Test  Find the optimal way to run Progress® on Linux  Test various ideas and theories  Have fun  Bunker 2005: Pre-release 10.1A 64-bit AMD Performance of Utilities Investigate “coma” problem Network speed effects

4. © 2006 Progress Software Corporation4 DB-14 Tales of the Bunker 2005 Why Linux?

5. © 2006 Progress Software Corporation5 DB-14 Tales of the Bunker 2005 Bunker 2005 Team  Gus Björklund, Wizard, Progress Software Progress User since 1989  John Harlow, President of BravePoint Progress User since 1984  Dan Foreman Progress User since 1984  Rich Banville, Fellow, Progress Software Progress User since 1993

6. © 2006 Progress Software Corporation6 DB-14 Tales of the Bunker 2005 The ATM Benchmark Environment

7. © 2006 Progress Software Corporation7 DB-14 Tales of the Bunker 2005 The ATM Benchmark  Simulates teller machine transactions deposit or withdrawal heavy database update workload  Each transaction does 3 fetches, 3 updates, 1 create –retrieve and update account, branch, and teller rows –create a history row  Run “n” transaction generators concurrently for fixed time period count total number of transactions performed

8. © 2006 Progress Software Corporation8 DB-14 Tales of the Bunker 2005 Test Database (logical) TableNumber of Rows Account80,000,000 (100 bytes each) Teller80,000 (100 bytes each) Branch8,000 (100 bytes each) Historyadd 1 per transaction (50 bytes each)

9. © 2006 Progress Software Corporation9 DB-14 Tales of the Bunker 2005 Test Database (physical) Total size12 gigabytes (data) Data extent size2,000,000 (2 GB) Data extent count6 Data block size4096 bytes Rows per block64 Data Areas1, Type II Data cluster size512 blocks (2 megabytes)

10. © 2006 Progress Software Corporation10 DB-14 Tales of the Bunker 2005 Test Database (other info) Account table data9.2 gigabytes Branch table data922 kilobytes Teller table data9.3 megabytes Indexes691 megabytes RM blocks2,669,712 Index blocks178,463 Free blocks150,682

11. © 2006 Progress Software Corporation11 DB-14 Tales of the Bunker 2005 Equipment

12. © 2006 Progress Software Corporation12 DB-14 Tales of the Bunker 2005 Server 1: hostname “uniblab” Operating SystemSuSE Linux Enterprise Server 9 Cpus2 x 2.8 GHz Intel Xeon MotherboardIBM xSeries Memory4 gigabytes Disk drives on controller6 x n GB 10,000 rpm USCSI

13. © 2006 Progress Software Corporation13 DB-14 Tales of the Bunker 2005 Server 3: hostname “hal” Operating SystemSuSE Linux 10.0 Cpus2 x 2.4 GHz Intel Xeon Memory2 gigabytes Disk drives on motherboard1 x n GB 7200 rpm SATA Disk drives on controller6 x n GB 7200 rpm SATA

14. © 2006 Progress Software Corporation14 DB-14 Tales of the Bunker 2005 Server 3: hostname “jumbo” Operating SystemSuSE Linux 10.0 Cpus2 x AMD 64-bit 2.0 GHz Dual Core MotherboardAsus K8N-DL Disk controller3Ware 9500S-8 Memory6 gigabytes Disk drives on motherboard2 x 160 GB Maxtor 7200 rpm Ultra IDE Disk drives on controller8 x 164 GB Hitachi Deskstar 7200 rpm SATA II

15. © 2006 Progress Software Corporation15 DB-14 Tales of the Bunker 2005 AMD-64 Hyper-Transport Design

16. © 2006 Progress Software Corporation16 DB-14 Tales of the Bunker 2005 Bunker Network Components  3 server machines  Netgear GS 105 Gigabit switch  SMC Barricade WAP  LinkSys WVC11b “bunker cam”  Various laptops running Windoze Linux Mac OS X 10.3

17. © 2006 Progress Software Corporation17 DB-14 Tales of the Bunker 2005 Bunker Network (partial) haluniblabjumbo gigabit switch wapbunker Cam Internet router laptops servers

18. © 2006 Progress Software Corporation18 DB-14 Tales of the Bunker 2005 Other Equipment

19. © 2006 Progress Software Corporation19 DB-14 Tales of the Bunker 2005 Past Results

20. © 2006 Progress Software Corporation20 DB-14 Tales of the Bunker 2005 Lessons from Past Bunkers  Type II Data Areas are faster  Don’t use Reiser File System  Use EXT3 or XFS File System  Don’t use the Anticipatory Scheduler Deadline or CFQ is better  2.6 Kernel is faster than 2.4 Kernel  For RAID 10, the Largest Possible Stripe Size was always the fastest, both Software & Hardware Striping  Very good performance at low cost

21. © 2006 Progress Software Corporation21 DB-14 Tales of the Bunker 2005 This Year’s (October 2005) Bunker Results

22. © 2006 Progress Software Corporation22 DB-14 Tales of the Bunker 2005 Setup Results db load time30 minutes db load rate2.7 million rows per minute dbanalysis3 min 55 sec idxbuild (-threads 1)8 min 54 sec idxbuild scratch1,129,764 KB prorest (from disk)9 minutes Database is about 11 gigabytes

23. © 2006 Progress Software Corporation23 DB-14 Tales of the Bunker 2005 Baseline Server Configuration  Data extents on striped array, BI log on own disk  BI cluster size:16384  BI blocksize: 16  Server options: –-n 200 -L 10240 –-B 64000 –-spin 50000 –-bibufs 32  Page writers: 4  BI writer: yes  AI writer: no

24. © 2006 Progress Software Corporation24 DB-14 Tales of the Bunker 2005 Linux I/O schedulers - 64-bit AMD SuSE 10 What do we learn from this?

25. © 2006 Progress Software Corporation25 DB-14 Tales of the Bunker 2005 Dump/Load  -index 0 Option on Binary Dump Excellent Performance Improvement But order of records may not be what you want

26. © 2006 Progress Software Corporation26 DB-14 Tales of the Bunker 2005 Dump/Load with -RO  OpenEdge® 10 -RO: Faster than V9 V9 with -RO: faster than without  OpenEdge 10 with -RO: performance same as without  OpenEdge 10 -RO: Clients now write entries in the.lg file

27. © 2006 Progress Software Corporation27 DB-14 Tales of the Bunker 2005 Dump/Load  Logical Scatter Factor is very important Performance Difference of 400% to 1000%

28. © 2006 Progress Software Corporation28 DB-14 Tales of the Bunker 2005 Remote Clients  -Ma The Lower Value, the Better the Performance  -Mm No Negative Impact on ATM Benchmarks

29. © 2006 Progress Software Corporation29 DB-14 Tales of the Bunker 2005 Network Speed

30. © 2006 Progress Software Corporation30 DB-14 Tales of the Bunker 2005 “Coma” Problem  We have experienced this problem in every Bunker Test  We still don’t know what’s wrong  A customer on RH AS 4 Kernel: 2.6.9-5.Elsmp reports problem solved - for him  There are an infinite number of things and combinations of things that can be changed in the kernel  We need to do some work with the “aggressiveness” of the APWs to help… but also more testing

31. © 2006 Progress Software Corporation31 DB-14 Tales of the Bunker 2005 Coma problem: -directio helps

32. © 2006 Progress Software Corporation32 DB-14 Tales of the Bunker 2005 We still think about this problem

33. © 2006 Progress Software Corporation33 DB-14 Tales of the Bunker 2005 64-Bit  We saw No difference in general performance between 32 and 64 bit Progress

34. © 2006 Progress Software Corporation34 DB-14 Tales of the Bunker 2005 Strange Problem  WEIRD PROBLEM ON ONE MACHINE:  The bigger –B, the lower the TPS rate True with both 32 and 64 bit Progress/Linux Could be caused by: –Enterprise versus Desktop version of Linux –10.1A Beta problem –SUSE Linux 10 issue (unsupported OS) –Something else –HyperTransport Effects –All the above  DID NOT OCCUR ON OTHER MACHINES  HAVE NOT SEEN AGAIN

35. © 2006 Progress Software Corporation35 DB-14 Tales of the Bunker 2005 TPS vs Response Time Avg Response time 0.2 seconds

36. © 2006 Progress Software Corporation36 DB-14 Tales of the Bunker 2005 V10.0B versus V10.1A Beta  No Difference in general performance

37. © 2006 Progress Software Corporation37 DB-14 Tales of the Bunker 2005 SATA versus SCSI  SCSI was faster  SATA is less expensive  Beware: desktop drives not rated for 24x7 operation

38. © 2006 Progress Software Corporation38 DB-14 Tales of the Bunker 2005 Online backup time workload

39. © 2006 Progress Software Corporation39 DB-14 Tales of the Bunker 2005 Online backup rate (approximate) workload

40. © 2006 Progress Software Corporation40 DB-14 Tales of the Bunker 2005 Online backup performance impact 150 user atm workload

41. © 2006 Progress Software Corporation41 DB-14 Tales of the Bunker 2005 Adding extent online: elapsed time Add 2 GB extent on same disk array operationno users150 users create extent18 seconds35 seconds enable extent0 seconds

42. © 2006 Progress Software Corporation42 DB-14 Tales of the Bunker 2005 Adding extent online: performance impact 150 user workload, add 2 GB data extent online, Extent on same striped array as other extents

43. © 2006 Progress Software Corporation43 DB-14 Tales of the Bunker 2005 Secret Bunker Web Pages  March 2002 & October 2002 http://www.myfloridacottage.com/benchmark.html  April 2004 http://www.myfloridacottage.com/bunker3.html  Oct 2005 http://www.myfloridacottage.com/bunker4/

44. © 2006 Progress Software Corporation44 DB-14 Tales of the Bunker 2005 Join Us in the Bunker  Film

45. © 2006 Progress Software Corporation45 DB-14 Tales of the Bunker 2005 Also see Gus’s RDBMS Tuning Guide on conference CD Want Answers

46. © 2006 Progress Software Corporation46 DB-14 Tales of the Bunker 2005