DB-14: Tales of the Bunker Gus Björklund, Progress Software Corporation John Harlow, Bravepoint, Inc. Dan Foreman, Bravepoint, Inc. Rich Banville, Progress Software Corporation

© 2006 Progress Software Corporation3 DB-14 Tales of the Bunker 2005 Goals of the Bunker Test  Find the optimal way to run Progress® on Linux  Test various ideas and theories  Have fun  Bunker 2005: Pre-release 10.1A 64-bit AMD Performance of Utilities Investigate “coma” problem Network speed effects

© 2006 Progress Software Corporation4 DB-14 Tales of the Bunker 2005 Why Linux?

© 2006 Progress Software Corporation5 DB-14 Tales of the Bunker 2005 Bunker 2005 Team  Gus Björklund, Wizard, Progress Software Progress User since 1989  John Harlow, President of BravePoint Progress User since 1984  Dan Foreman Progress User since 1984  Rich Banville, Fellow, Progress Software Progress User since 1993

© 2006 Progress Software Corporation6 DB-14 Tales of the Bunker 2005 The ATM Benchmark Environment

© 2006 Progress Software Corporation7 DB-14 Tales of the Bunker 2005 The ATM Benchmark  Simulates teller machine transactions deposit or withdrawal heavy database update workload  Each transaction does 3 fetches, 3 updates, 1 create –retrieve and update account, branch, and teller rows –create a history row  Run “n” transaction generators concurrently for fixed time period count total number of transactions performed

© 2006 Progress Software Corporation8 DB-14 Tales of the Bunker 2005 Test Database (logical) TableNumber of Rows Account80,000,000 (100 bytes each) Teller80,000 (100 bytes each) Branch8,000 (100 bytes each) Historyadd 1 per transaction (50 bytes each)

© 2006 Progress Software Corporation9 DB-14 Tales of the Bunker 2005 Test Database (physical) Total size12 gigabytes (data) Data extent size2,000,000 (2 GB) Data extent count6 Data block size4096 bytes Rows per block64 Data Areas1, Type II Data cluster size512 blocks (2 megabytes)

© 2006 Progress Software Corporation10 DB-14 Tales of the Bunker 2005 Test Database (other info) Account table data9.2 gigabytes Branch table data922 kilobytes Teller table data9.3 megabytes Indexes691 megabytes RM blocks2,669,712 Index blocks178,463 Free blocks150,682

© 2006 Progress Software Corporation11 DB-14 Tales of the Bunker 2005 Equipment

© 2006 Progress Software Corporation12 DB-14 Tales of the Bunker 2005 Server 1: hostname “uniblab” Operating SystemSuSE Linux Enterprise Server 9 Cpus2 x 2.8 GHz Intel Xeon MotherboardIBM xSeries Memory4 gigabytes Disk drives on controller6 x n GB 10,000 rpm USCSI

© 2006 Progress Software Corporation13 DB-14 Tales of the Bunker 2005 Server 3: hostname “hal” Operating SystemSuSE Linux 10.0 Cpus2 x 2.4 GHz Intel Xeon Memory2 gigabytes Disk drives on motherboard1 x n GB 7200 rpm SATA Disk drives on controller6 x n GB 7200 rpm SATA

© 2006 Progress Software Corporation14 DB-14 Tales of the Bunker 2005 Server 3: hostname “jumbo” Operating SystemSuSE Linux 10.0 Cpus2 x AMD 64-bit 2.0 GHz Dual Core MotherboardAsus K8N-DL Disk controller3Ware 9500S-8 Memory6 gigabytes Disk drives on motherboard2 x 160 GB Maxtor 7200 rpm Ultra IDE Disk drives on controller8 x 164 GB Hitachi Deskstar 7200 rpm SATA II

© 2006 Progress Software Corporation15 DB-14 Tales of the Bunker 2005 AMD-64 Hyper-Transport Design

© 2006 Progress Software Corporation16 DB-14 Tales of the Bunker 2005 Bunker Network Components  3 server machines  Netgear GS 105 Gigabit switch  SMC Barricade WAP  LinkSys WVC11b “bunker cam”  Various laptops running Windoze Linux Mac OS X 10.3

© 2006 Progress Software Corporation17 DB-14 Tales of the Bunker 2005 Bunker Network (partial) haluniblabjumbo gigabit switch wapbunker Cam Internet router laptops servers

© 2006 Progress Software Corporation18 DB-14 Tales of the Bunker 2005 Other Equipment

© 2006 Progress Software Corporation19 DB-14 Tales of the Bunker 2005 Past Results

© 2006 Progress Software Corporation20 DB-14 Tales of the Bunker 2005 Lessons from Past Bunkers  Type II Data Areas are faster  Don’t use Reiser File System  Use EXT3 or XFS File System  Don’t use the Anticipatory Scheduler Deadline or CFQ is better  2.6 Kernel is faster than 2.4 Kernel  For RAID 10, the Largest Possible Stripe Size was always the fastest, both Software & Hardware Striping  Very good performance at low cost

© 2006 Progress Software Corporation21 DB-14 Tales of the Bunker 2005 This Year’s (October 2005) Bunker Results

© 2006 Progress Software Corporation22 DB-14 Tales of the Bunker 2005 Setup Results db load time30 minutes db load rate2.7 million rows per minute dbanalysis3 min 55 sec idxbuild (-threads 1)8 min 54 sec idxbuild scratch1,129,764 KB prorest (from disk)9 minutes Database is about 11 gigabytes

© 2006 Progress Software Corporation23 DB-14 Tales of the Bunker 2005 Baseline Server Configuration  Data extents on striped array, BI log on own disk  BI cluster size:16384  BI blocksize: 16  Server options: –-n 200 -L –-B –-spin –-bibufs 32  Page writers: 4  BI writer: yes  AI writer: no

© 2006 Progress Software Corporation24 DB-14 Tales of the Bunker 2005 Linux I/O schedulers - 64-bit AMD SuSE 10 What do we learn from this?

© 2006 Progress Software Corporation25 DB-14 Tales of the Bunker 2005 Dump/Load  -index 0 Option on Binary Dump Excellent Performance Improvement But order of records may not be what you want

© 2006 Progress Software Corporation26 DB-14 Tales of the Bunker 2005 Dump/Load with -RO  OpenEdge® 10 -RO: Faster than V9 V9 with -RO: faster than without  OpenEdge 10 with -RO: performance same as without  OpenEdge 10 -RO: Clients now write entries in the.lg file

© 2006 Progress Software Corporation27 DB-14 Tales of the Bunker 2005 Dump/Load  Logical Scatter Factor is very important Performance Difference of 400% to 1000%

© 2006 Progress Software Corporation28 DB-14 Tales of the Bunker 2005 Remote Clients  -Ma The Lower Value, the Better the Performance  -Mm No Negative Impact on ATM Benchmarks

© 2006 Progress Software Corporation29 DB-14 Tales of the Bunker 2005 Network Speed

© 2006 Progress Software Corporation30 DB-14 Tales of the Bunker 2005 “Coma” Problem  We have experienced this problem in every Bunker Test  We still don’t know what’s wrong  A customer on RH AS 4 Kernel: Elsmp reports problem solved - for him  There are an infinite number of things and combinations of things that can be changed in the kernel  We need to do some work with the “aggressiveness” of the APWs to help… but also more testing

© 2006 Progress Software Corporation31 DB-14 Tales of the Bunker 2005 Coma problem: -directio helps

© 2006 Progress Software Corporation32 DB-14 Tales of the Bunker 2005 We still think about this problem

© 2006 Progress Software Corporation33 DB-14 Tales of the Bunker Bit  We saw No difference in general performance between 32 and 64 bit Progress

© 2006 Progress Software Corporation34 DB-14 Tales of the Bunker 2005 Strange Problem  WEIRD PROBLEM ON ONE MACHINE:  The bigger –B, the lower the TPS rate True with both 32 and 64 bit Progress/Linux Could be caused by: –Enterprise versus Desktop version of Linux –10.1A Beta problem –SUSE Linux 10 issue (unsupported OS) –Something else –HyperTransport Effects –All the above  DID NOT OCCUR ON OTHER MACHINES  HAVE NOT SEEN AGAIN

© 2006 Progress Software Corporation35 DB-14 Tales of the Bunker 2005 TPS vs Response Time Avg Response time 0.2 seconds

© 2006 Progress Software Corporation36 DB-14 Tales of the Bunker 2005 V10.0B versus V10.1A Beta  No Difference in general performance

© 2006 Progress Software Corporation37 DB-14 Tales of the Bunker 2005 SATA versus SCSI  SCSI was faster  SATA is less expensive  Beware: desktop drives not rated for 24x7 operation

© 2006 Progress Software Corporation38 DB-14 Tales of the Bunker 2005 Online backup time workload

© 2006 Progress Software Corporation39 DB-14 Tales of the Bunker 2005 Online backup rate (approximate) workload

© 2006 Progress Software Corporation40 DB-14 Tales of the Bunker 2005 Online backup performance impact 150 user atm workload

© 2006 Progress Software Corporation41 DB-14 Tales of the Bunker 2005 Adding extent online: elapsed time Add 2 GB extent on same disk array operationno users150 users create extent18 seconds35 seconds enable extent0 seconds

© 2006 Progress Software Corporation42 DB-14 Tales of the Bunker 2005 Adding extent online: performance impact 150 user workload, add 2 GB data extent online, Extent on same striped array as other extents

© 2006 Progress Software Corporation43 DB-14 Tales of the Bunker 2005 Secret Bunker Web Pages  March 2002 & October  April  Oct

© 2006 Progress Software Corporation44 DB-14 Tales of the Bunker 2005 Join Us in the Bunker  Film

© 2006 Progress Software Corporation45 DB-14 Tales of the Bunker 2005 Also see Gus’s RDBMS Tuning Guide on conference CD Want Answers

© 2006 Progress Software Corporation46 DB-14 Tales of the Bunker 2005