1. PROGRESS S O F T W A R E Linux And The Progress RDBMS Gus Björklund (gus@progress.com) Wizard Progress Software PUG Challenge 2002, Veldhoven, the Netherlands

2. PROGRESS S O F T W A R E Engine Crew

3. © 2002, Progress Software Corporation 3 PROGRESS S O F T W A R E Abstract The Linux operating system can be the right choice for many Progress RDBMS installations. This talk describes how an Intel-based system can be a highly cost-effective and powerful database server running the Progress RDBMS on RedHat Linux 7.2 on a powerful but inexpensive computer. We will show how to configure the system, what to do, and some things to avoid. Informal benchmark results from the Spring of 2002 are used to illustrate the performance of various configurations. Time: 90 minutes

4. © 2002, Progress Software Corporation 4 PROGRESS S O F T W A R E Please ask questions if I do not explain something clearly

5. © 2002, Progress Software Corporation 5 PROGRESS S O F T W A R E Spring 2002 Linux Benchmark Team  Bravepoint  Dan Foreman  John Harlow  Progress  Gus Björklund

6. © 2002, Progress Software Corporation 6 PROGRESS S O F T W A R E Goals  Prove that Linux is stable and viable in the real world  Prove that price/performance is excellent  Produce configuration recommendations  And finally…...

7. © 2002, Progress Software Corporation 7 PROGRESS S O F T W A R E

8. © 2002, Progress Software Corporation 8 PROGRESS S O F T W A R E Why Linux ?  Commodity Priced Hardware  Inexpensive Operating System  Linux is no longer a toy  Currently being used in production (details to follow) in mission critical applications  Excellent alternative to Microsoft  Believers in Open Source  We don’t drink Microsoft Kool-Aid

9. © 2002, Progress Software Corporation 9 PROGRESS S O F T W A R E Why Not Linux ?  The constant temptation to apply the “patch of the day”  Credibility (the gap between perception of reality and reality)  Track record is relatively short

10. © 2002, Progress Software Corporation 10 PROGRESS S O F T W A R E Why Not Linux  It’s not like an OS where you call up your friendly 24 hr/day support line and they stick with fixing your problem; we posted a note to one of the Linux lists during our testing and never got a response; There are two ways to go:  the slow, plodding, conservative method  the complex, inter-dependent, patch-of- the-day method

11. © 2002, Progress Software Corporation 11 PROGRESS S O F T W A R E History  August 25, 1991: Linus Torvalds posts to comp.os.minix that he has his experimental kernel running gcc and bash.  “just a hobby, won’t be big and professional”  Version 0.01 sources posted September 1991  1.0 stable kernel release was March 1994  SCO OpenServer binaries of Progress were being run on Linux as early as 1995  Not supported by PSC  First Progress release for Linux (RedHat 6.2) was Version 8.3C in Spring 2000

12. © 2002, Progress Software Corporation 12 PROGRESS S O F T W A R E Example Production Linux Site  Transplatinum  Financial services provider for large trucking companies  Progress V9.1B  Linux 2.2.19-6.2.1  DB Size 12GB  137 Users  Only problem encountered: “getting the correct Linux patch combinations installed”

13. © 2002, Progress Software Corporation 13 PROGRESS S O F T W A R E Benchmark Hardware In Zero Gravity

14. © 2002, Progress Software Corporation 14 PROGRESS S O F T W A R E Benchmark Server Hardware  Disks - 10 IBM Deskstar IDE (not SCSI) disk drives, 60 gigabytes each  3Ware 7810 Disk Controller with H/W RAID  Memory - 2 gigabytes, PC133, SDRAM  CPUs - 2, 1 gigahertz Pentium III  ASUS Motherboard

15. © 2002, Progress Software Corporation 15 PROGRESS S O F T W A R E Hardware Cost (August 2001) ItemPrice Server Case$528 Hot Swap Power Supply$338 10 IBM Deskstar 60 GB IDE disks$1890 8 Hot swap cases for disks$200 3Ware 7810 RAID controller$385 Motherboard, RAM, CD, Floppy, Video$891 Shipping$161 Total$4413

16. © 2002, Progress Software Corporation 16 PROGRESS S O F T W A R E Hardware Cost (1991) Sequent SMP Unix System ItemPrice 16 x 20 megahertz 80386 cpu, 64 MB RAM, cabinet $318,500 176 megabytes additional RAM$195,200 21 Fujitsu Eagle 200 MB ? disks$246,000 Dynix/PTX Operating System$46,000 Installation$8,090 Miscellaneous items$23,500 Total$837,290

17. © 2002, Progress Software Corporation 17 PROGRESS S O F T W A R E Then Versus Now  Result Then:  170 tps$4,925 per tps  Result Now:  580 tps$7.60 per tps 3.4 x more performance 1 / 648 the price per tps

18. © 2002, Progress Software Corporation 18 PROGRESS S O F T W A R E Software Details  Linux 2.4.18 Kernel  Stock except for the RAID array drivers  Progress V9.1C09

19. © 2002, Progress Software Corporation 19 PROGRESS S O F T W A R E Multi-million dollar benchmarking lab in a secret location

20. © 2002, Progress Software Corporation 20 PROGRESS S O F T W A R E Benchmark Laboratory Underground Bunker

21. © 2002, Progress Software Corporation 21 PROGRESS S O F T W A R E Benchmark Team (Partial)

22. © 2002, Progress Software Corporation 22 PROGRESS S O F T W A R E Benchmarks  Lies, True Lies, and Benchmarks  ATM benchmark  Didn’t compare performance to NT, W2K, or any other Intel Operating System

23. © 2002, Progress Software Corporation 23 PROGRESS S O F T W A R E ATM Benchmark  Simulates ATM withdrawal transaction  read, update account  read, update branch  read, update teller  create history  Execute as many times as possible in given time  Produces heavy update workload

24. © 2002, Progress Software Corporation 24 PROGRESS S O F T W A R E ATM Database  4 tables  account: 20,000,000 records  branch: 20,000 records  teller: 2,000 records  history: create 1 per transaction  3.2 gigabyte total size  2 extents, 1,600 megabytes each  2,300 megabytes initial data  175 megabytes indexes

25. © 2002, Progress Software Corporation 25 PROGRESS S O F T W A R E Creating The 3.2 GB Database  prostrct create to allocate space  about 2 minutes  Creating the initial records via 4GL  35 min  procopy to make backup  10 minutes, including creating extents

26. © 2002, Progress Software Corporation 26 PROGRESS S O F T W A R E Other Times  Index rebuild (eventually)  9 minutes, 10 seconds  proutil atm -C idxbuild all -TB 31 -TM 32 -B 250 -T /bi -t  sort file size: 381,086,720 bytes  dbanalysis  6 minutes

27. © 2002, Progress Software Corporation 27 PROGRESS S O F T W A R E Documentation

28. © 2002, Progress Software Corporation 28 PROGRESS S O F T W A R E Baseline Run  Database size  3 gigabytes  Everything on one disk  8k database block size but otherwise untuned “out-of-the box installation”

29. © 2002, Progress Software Corporation 29 PROGRESS S O F T W A R E Baseline Run  Database size  3 gigabytes  4 tables n account, branch, teller, history  Everything on one disk  8k database block size but otherwise untuned “out-of-the box installation”  Result: 30 tps (4 users)

30. © 2002, Progress Software Corporation 30 PROGRESS S O F T W A R E Top 8 Progress Tuning Items  1.BI on a separate disk  2.-spin 50,000  3.-B 32000  Later we tried 64000  4.BI Cluster Size (-bi) 16 mb

31. © 2002, Progress Software Corporation 31 PROGRESS S O F T W A R E Top 8 Progress Tuning Items  5.Page Writers (4)  6.BI Writer  7.BI Buffers (25+)  8.8K Database Block Size

32. © 2002, Progress Software Corporation 32 PROGRESS S O F T W A R E Top 8 Progress Tuning Items  5.Page Writers (4)  6.BI Writer  7.BI Buffers (25+)  8.8K Database Block Size  Results before:30 tps (4 users)  Results after:61 tps (4 users)

33. © 2002, Progress Software Corporation 33 PROGRESS S O F T W A R E Linux Changes  Reiser FS  some of the best and worst numbers  too erratic so we didn’t continue  EXT2 versus EXT3  no difference observed  noatime  no difference observed  Larger shared memory segments (32mb default raised to128 mb)  2% gain

34. © 2002, Progress Software Corporation 34 PROGRESS S O F T W A R E Additional Progress Changes  -semsets  no statistically significant difference  Index rebuild of the test database  3% gain

35. © 2002, Progress Software Corporation 35 PROGRESS S O F T W A R E Additional Progress Changes  Doubled -B from 32000 to 64000 (8k blocks)  Before:557 tps (54 users)  After:581 tps (58 users)

36. © 2002, Progress Software Corporation 36 PROGRESS S O F T W A R E Additional Progress Changes  File System Block Size/DB Block Size rated from best to worst:  4k/8k  4k/4k  1k/8k - not tested n prostrct create took too long  1k/1k- not tested n prostrct create took too long

37. © 2002, Progress Software Corporation 37 PROGRESS S O F T W A R E Disk Array Changes  JBOD  8 disks  479 tps (24 users)  RAID 0  Striped 8 disks, 64k stripe size, bi on stripe set  339 tps (26 users)  Moved BI file to internal (non-RAID) drives  481 tps (32 users)

38. © 2002, Progress Software Corporation 38 PROGRESS S O F T W A R E Disk Array Changes  RAID 0  8 disks; just striping  -B 64000, index rebuild, raise shmmax  581 tps (58 users)  Stripe & Mirror  3 x 2 for data disks  1 pair for BI  304 tps (60 users)

39. © 2002, Progress Software Corporation 39 PROGRESS S O F T W A R E Disk Setups Compared Configurationtps% of base 1 60 GB internal IDE disk (base)61100 JBOD, 8 disks479785 RAID 0, 8 disks, data + bi339555 RAID 0, 8 data + separate bi disk481788 RAID 0, tune + separate bi disk581952 RAID 10, 3 pair data, 1 pair bi304498

40. © 2002, Progress Software Corporation 40 PROGRESS S O F T W A R E Conventional Wisdom  There’s no such thing!  Direct I/O parameter reduced performance (551  118) but chopped off the spikes in the longest transaction duration; resorted to manual syncing similar to pre-directio days  Enabling On-board disk cache DECREASED Performance (506  474)  RAID 0 Stripe Size 1mb was much better than 64k

41. © 2002, Progress Software Corporation 41 PROGRESS S O F T W A R E Cost Of After Imaging  Before:289 tps (92 users)  After:278 tps (82 users)  3.8% reduction

42. © 2002, Progress Software Corporation 42 PROGRESS S O F T W A R E Constraints  Time - infinite number of benchmarking possibilities  The unexpected: 3 hours to configure the RAID array from JBOD to RAID 10  Ran out of beer on the last night  Occasional Disagreement between the Members of the Benchmarking Team  BI Buffers  What do we do next?  Where do we eat tonight?

43. © 2002, Progress Software Corporation 43 PROGRESS S O F T W A R E Linux Virtual Memory  bdflush  It’s in flux  Hurriedly overhauled in 2.4.10  Will be overhauled again in 2.4.19, soon  Direct I/O does not produce results similar to other Unixen

44. © 2002, Progress Software Corporation 44 PROGRESS S O F T W A R E Miscellaneous Stuff  Can’t control the amount of OS Cache  at least, we could not

45. © 2002, Progress Software Corporation 45 PROGRESS S O F T W A R E What we Didn’t Do  RAID 5 - because as you know it’s Evil  Software striping & mirroring  Raw partitions  Non-standard file systems (JFS, etc)  2.5 kernel  Timeslice parameters  NAS  Storage Areas  Records per Block

46. © 2002, Progress Software Corporation 46 PROGRESS S O F T W A R E Yes, but what about Windows

47. © 2002, Progress Software Corporation 47 PROGRESS S O F T W A R E Digression: Linux Vs. Windows Performance  Progress 9.1A05  Windows 2000 Advanced Server @$3,500 USD  RedHat Linux 6.2, 2.2.14 kernel  Same exact hardware for both  IBM Netfinity 8500, 8 x 550 mhz Pentium III  8 gigabytes RAM  40 x 9 Gigabyte SCSI disks  32 megabytes of disk cache In a previous benchmark, we compared Linux against Windows

48. © 2002, Progress Software Corporation 48 PROGRESS S O F T W A R E Linux Vs Windows Performance Results (9.1A05)  Windows:3,655 tps  Linux4,508 tps Linux was 23 % faster Linux had more idle cpu ( ~ 15 %)

49. © 2002, Progress Software Corporation 49 PROGRESS S O F T W A R E Conclusions  Linux provides excellent performance  Could probably run 100 MFG/PRO users  System was easy to set up, easy to use  System cost was very low  We had no way to backup the system !!  493 GB of disk in the array  Tape robot could cost $12,000 or more  Better than Windows on Intel

50. © 2002, Progress Software Corporation 50 PROGRESS S O F T W A R E What Should We Test Next Time ?  Next Linux Benchmark Oct 28, 2002  Should we  compare Linux vs Windows with 9.1D?  compare AMD vs Intel ?  compare filesystems in detail?  try BIOS “tweaking”?  Ideas?

51. © 2002, Progress Software Corporation 51 PROGRESS S O F T W A R E Vragen