Download presentation
Presentation is loading. Please wait.
1
Some More Database Performance Knobs North American PUG Challenge
Richard Banville Software Fellow OpenEdge Development
2
1 2 3 4 5 Agenda LRU (again) Networking: Message Capacity
Networking: Resource Usage 4 Index Rebuild 5 Summary
3
1 2 3 4 5 Agenda LRU (again) Networking: Message Capacity
Networking: Resource Usage 4 Index Rebuild 5 Summary
4
LRU (again) Least Recent Most Recent RM Block T1 IX Block I1
Replacement policy of database buffer pool Maintains working set of data buffers Just a linked list – a shared data structure Changes made orderly by LRU Latch Replace buffer at LRU end with newly read block from disk Just a linked list meandering though the buffer pool
5
LRU (again) Least Recent Most Recent RM Block T1 IX Block I1
Pros – proficient block usage predictor Maintains high buffer pool hit ratio Cons – housekeeping costs Single threads access to buffer pool (even if for an instant) High activity, relatively high nap rate Managing LRU: Private read only buffers: -Bp –BpMax (not w/-lruskips until 10.2b07) Alternate buffer pool: –B2 New: -lruskips lru2skips Just a linked list meandering though the buffer pool
6
LRU (again) Least Recent Most Recent RM Block T1 IX Block I1
Find first T1. Just a linked list meandering though the buffer pool
7
LRU (again) Least Recent Most Recent RM Block T1 IX Block I1
Find first T1. RM Block T1 RM Block T3 IX Block I3 IX Block I1 RM Block T3 IX Block I3 IX Block I1 RM Block T1 Just a linked list meandering though the buffer pool
8
LRU (again) Least Recent Most Recent RM Block T3 IX Block I3
Find first T1. (again) RM Block T3 IX Block I3 RM Block T1 IX Block I1 RM Block T3 IX Block I3 IX Block I1 RM Block T1 Just a linked list meandering though the buffer pool What about … For each T1: end. For each w/many tables. For each w/many tables, many users.
9
Location, location, location
Least Recent Most Recent With –B 1,000,000 What does it take to evict from the buffer pool? What does it take to go from MRU to LRU? Do we need MRU on EACH access then? I think not.
10
Improving Concurrency
Least Recent Most Recent -lruskips <n> LRU and LFU combined Small numbers make a BIG difference Monitor OS Read I/Os and LRU latch contention Adjust online via _Startup. _Startup-LRU-Skips VST field Adjust online via promon R&D -> 4. Administrative Functions ... -> 4. Adjust LRU force skips
11
Performance – 10.2b06 & -lruskips
Re-iterate - read performance improvement for high volume/contention no-lock read situation Performance starts to degrade at about 140 users. ~39% # Users
12
Performance – 10.2b06 & -lruskips (250 users)
Note change in LRU latch waits vs buffer latch waits
13
Performance – 10.2b06 & -lruskips (250 users, big db)
Note focus now is on LRU and BHT (not buf)
14
Performance – 10.2b06 & -lruskips (big db)
~52% ~15% ~44% Re-iterate - read performance improvement for high volume/contention no-lock read situation Performance starts to degrade at about 140 users. # Users
15
Conclusions -lruskips can eliminate the LRU bottleneck
LRU isn’t the last bottleneck Overall improvement relative to other contention Data access limited by buffer level contention Table scans over small tables have more buffer contention than large tables Application changes can improve performance too!
16
1 2 3 4 5 Agenda LRU (again) Networking: Message Capacity
Networking: Resource Usage 4 Index Rebuild 5 Summary
17
Networking Control Philosophy: Throughput by keeping server busy without remote client waits! Process based control -Ma, -Mn, -Mi Controls the order users are assigned to servers -PendCondTime Resource based control -Mm <n> Maximum size of network message Client & server startup New tuning knobs – resource based control Alleviate excessive system CPU usage by network layer Control record data stuffed in a network message Applicable for “prefetch” queries Polling for Unix only Message stuffing supported for both windows and unixes
18
Networking – Prefetch Query
No-lock query with guaranteed forward motion or scrolling Multiple records stuffed into single network message Browsed static and preselected queries scrolling by default FOR EACH customer NO-LOCK: …. end. DO PRESELECT EACH customer NO-LOCK: …. end. Static queries are associated with class files only. Note that the third query MUST include the SCROLLING options define query cust-q for customer SCROLLING. open query cust-q FOR EACH customer NO-LOCK. repeat: get next cust-q. end.
19
Server Network Message Processing Loop
Polling for Unix only Message stuffing supported for both windows and unixes
20
Server Network Message Processing Loop
Polling for Unix only Message stuffing supported for both windows and unixes
21
Server Network Message Processing Loop
-NmsgWait What’s new: Polling for Unix only Message stuffing supported for both windows and unixes
22
Server Network Message Processing Loop
Polling for Unix only Message stuffing supported for both windows and unixes
23
Server Network Message Processing Loop
Polling for Unix only Message stuffing supported for both windows and unixes
24
Server Network Message Processing Loop
Poll() is system CPU intensive Polling for Unix only Message stuffing supported for both windows and Unixes 1ms = .001 mu => or 1000 mu = 1 ms 10 milliseconds to poll(0)! 10 microseconds to copy 1 record
25
Server Network Message Processing Loop
Potential side effects What’s new: -Nmsgwait – Network message wait time -prefetchPriority – give prefetching of records priority over polling for new requests -prefetchPriority
26
Server Network Message Processing Loop
Polling for Unix only Message stuffing supported for both windows and unixes
27
Process Waiting Network Message
28
Process Waiting Network Message
Non-prefetch query request
29
Process Waiting Network Message
1st record of a prefetch query request
30
Process Waiting Network Message
Secondary records of a prefetch query request: threshold not met default threshold is 16 records Default threshold is 16 records : 4096/16 = 256 bytes
31
Process Waiting Network Message
Secondary records of a prefetch query request: Client waiting Threshold met Send message
32
Process Waiting Network Message
What’s new: Increase network message fill rate: Improve TCP throughput Improve overall server performance Defaults have not changed Provides control for you Every deployment is different
33
Process Waiting Network Message
Disregard 1st record request check -prefetchDelay Threshold control – mention that first limit reached causes message to be sent. 0% disables it. Threshold control: # recs vs % full -prefetchNumRecs -prefetchFactor Potential side effects: Improved TCP/system performance Choppy behavior on remote client? NOTE: - -Mm size determines max Mm 4096 / 16 rec = 256 bytes
34
Altering Network Message Behavior
Promon Support (_Startup VST too!) Alter online R&D … 4. Administrative Functions … 7. Server Options … Server Options: 1. Server network message wait time: seconds 2. Delay first prefetch message: Enabled 3. Prefetch message fill percentage: % 4. Minimum records in prefetch message: 1000 5. Suspension queue poll priority: 7. Terminate a server Alter via _Startup vst
35
Performance – 10.2b06 & Networking changes
Re-iterate - read performance improvement for high volume/contention no-lock read situation Performance starts to degrade at about 140 users. ~212% ~32% # Users
36
1 2 3 4 5 Agenda LRU (again) Networking: Message Capacity
Networking: Resource Usage 4 Index Rebuild 5 Summary
37
Assumptions for best performance
Index data is segregated from table data Indexes & tables are in different storage areas You have enough disk space for sorting You understand the impact of CPU and memory consumption Process allowed to use available system resources
38
Index Rebuild Parameters - Overview
-TB sort block size (8K – 64K, note new limit) -datascanthreads # threads for data scan phase -TMB merge block size ( default -TB) -TF merge pool fraction of system memory (in %) -mergethreads # threads per concurrent sort group merging -threadnum # concurrent sort group merging -TM # merge buffers to merge each merge pass -rusage report system usage statistics -silent a bit quieter than before
39
Phases of Index Rebuild (“non-recoverable”)
Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Index Scan Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Data Scan/ Key Build Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Sort-Merge Read –TF or temp sort file Insert keys into index Formats new clusters; May raise HWM I/O Bound with little CPU Activity Index Key Insertion
40
Phases of Index Rebuild
Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Index Scan Area 9: Index scan (Type II) complete. Index area is scanned start to finish (single threaded) Block at a time with cluster hops Index blocks are put on free chain for the index Index Object is not deleted (to fix corrupt cluster or block chains) Order of operation: Blocks are read from disk, Blocks are re-formatted in memory Blocks are written to disk as –B is exhausted Causes I/O in other phases for block re-format Can be eliminated with manual area truncate where possible There is no start message for the index scan phase, only a complete message
41
Phases of Index Rebuild
Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Index Scan Data Scan/ Key Build Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Processing area 8 : (11463) Start 4 threads for the area. (14536) Area 8: Multi-threaded record scan (Type II) complete. Table data area is scanned start to finish (multi-threaded if –datascanthreads) Each thread processes next block in area (with cluster hops) Database re-opened by each thread in R/O mode Ensure file handle ulimits set high enough
42
… Data Scan/Key Build Thread reads next data block in data area
DB RM Block Thread reads next data block in data area Record Extract next record from data block and build index key (sort order) Key Sort Block Sort Block Insert key into sort block (-TB 8K thru 64K) Sort/merge full sort block into merge block. (-TMB -TB thru 64K) Merge Block Write merge block to –TF, overflow to temp (-TMB sized I/O) -TF .srt1 .srt2 …
43
Sort Groups: -SG 3 (note 8 is minimum)
Each index assigned a particular sort group (hashed index #) 1) -T /usr1/richb/temp/ Index 1 SG 1 .srt1 Index 4 2) <dbname>.srt /usr1/richb/temp/ Record Index 2 SG 2 .srt2 3) <dbname>.srt /usr1/richb/temp/ /usr1/richb/temp/ Index 3 SG 3 .srt3 4) <dbname>.srt /usr1/richb/temp/ /usr2/richb/temp/ /usr3/richb/temp/ Each group has its own sort file Sort file location 1. Sort files in same directory (I/O contention) 4. Sort files in different location Ensure enough space
44
Phases of Index Rebuild
Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Index Scan Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Data Scan/ Key Build Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Sort-Merge Sorting index group 3 Spawning 4 threads for merging of group 3. Sorting index group 3 complete.
45
Sort-Merge Phase Sort blocks in each sort group have been sorted and merged into a linked list of individual merge blocks stored in –TF and temp files. These merge blocks are further merged –TM# at a time to form new larger “runs” of sorted merge blocks. -TM# of these new “runs” are then merged to form even larger “runs” of sorted merge blocks. When there is only one very large “run” left, all the key entries in the sort group are in sorted order. Sorted! Note: even thought the illustration does not depict merges of –TM “runs”, they are always merged –TM at a time. It is just too difficult to depict given the screen real estate.
46
-threadnum vs -mergethreads
-TF .srt1 Thread 1 Merge phase group 1 -TF .srt2 Thread 2 Merge phase group 2 -TF .srt3
47
-threadnum vs -mergethreads
B-tree insertion occurs as soon as a sort group’s merge is completed. -TF .srt1 Thread 0 begins b-tree insertion concurrently. Thread 0 -TF .srt2 Thread 2 Merge phase group 2 -TF .srt3 Thread 1 Merge phase group 3
48
-threadnum vs -mergethreads
-threadnum 2 –mergethreads 3 Merge threads merge successive “runs” of merge blocks concurrently. Thread 3 -TF .srt1 Thread 1 Thread 4 Merge phase group 1 Thread 5 -TF .srt2 Thread 6 Thread 2 Merge phase group 2 Thread 7 Thread 8 -TF .srt3 Note: 8 actively running threads
49
-threadnum vs -mergethreads
-threadnum 2 –mergethreads 3 -TF .srt1 -TF .srt2 Thread 6 Thread 2 Merge phase group 2 Thread 7 Thread 8 -TF .srt3 Thread 3 Thread 1 Merge phase group 3 Thread 4 Thread 5
50
-threadnum vs -mergethreads
-threadnum 2 –mergethreads 3 B-tree insertion occurs as soon as a sort group’s merge is completed. -TF .srt1 Thread 0 begins b-tree insertion concurrently. Thread 0 -TF .srt2 Thread 6 Thread 2 Merge phase group 2 Thread 7 Thread 8 -TF .srt3 Thread 3 Thread 1 Merge phase group 3 Thread 4 Thread 5 Note: 9 actively running threads
51
Phases of Index Rebuild
Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Index Scan Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Data Scan/ Key Build Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Sort-Merge Index Key Insertion Read –TF or temp sort file Insert keys into index Formats new clusters; May raise HWM I/O Bound with little CPU Activity
52
Index Key Insertion Phase
Building index 11 (cust-num) of group 3 … Building of indexes in group 3 completed. Multi-threaded index sorting and building complete. Key entries from sorted merge blocks are inserted into b-tree Performed sequentially entry at a time, index at a time Leaf level insertion optimization (avoids b-tree scan) Leaf level written to disk as soon as full (since never revisited) Index B-tree Root Leaf Leaf Leaf Leaf level optimization internally referred to as “cxfast” insertion Write leaf when full DB
53
2085 Indexes were rebuilt. (11465) Index rebuild complete
2085 Indexes were rebuilt. (11465) Index rebuild complete. 0 error(s) encountered.
54
Index Rebuild - Tuning Truncate index only area if possible Parameters
.srt file Parameters -mergethreads: 2 or 4 and –threadnum 2 or 1 -datascanthreads: 1.5 * # CPUs -B 1024 –TF 80 (monitor physical memory paging) –TMB 64 –TB 64 –TM 32 –T: separate disk, RAM disk if not using -TF (no change) -rusage & -silent Double scan – how to avoid - removed
55
12 ½ hours 2 ½ hours 5X improvement!
Performance Numbers Elapsed Time 12 ½ hours 2 ½ hours 5X improvement! Cost of each phase (in secs)
56
1 2 3 4 5 Agenda LRU (again) Networking: Message Capacity
Networking: Resource Usage 4 Index Rebuild 5 Summary
57
Summary LRU Networking Index Rebuild Potential for a big win
Always room for improvement Us and you! Networking You now have more control With power comes responsibility Index Rebuild Big improvements if Your database is setup properly You provide system resources to index rebuild Hopefully you’ll never need it
58
? Questions
60
Performance – 10.2b06 & -lruskips (250 users)
Latches/sec
61
Performance – 10.2b06 & -lruskips (250 users, big db)
Latches/sec
62
Networking – Promon Support (VSTs too!)
No Prefetch Prefetch, default settings Activity: Servers Total Messages received Messages sent Bytes received Bytes sent Records received Records sent Queries received Time slices Activity: Servers Total Messages received Messages sent Bytes received Bytes sent Records received Records sent Queries received Time slices Prefetch, 90% capacity Prefetch, 90% cap. & delay Activity: Servers Total Messages received Messages sent Bytes received Bytes sent Records received Records sent Queries received Time slices Activity: Servers Total Messages received Messages sent Bytes received Bytes sent Records received Records sent Queries received Time slices
Similar presentations
![Dr. Kalpakis CMSC 661, Principles of Database Systems Index Structures [13]](/15/4658104/big_thumb.jpg "Dr. Kalpakis CMSC 661, Principles of Database Systems Index Structures [13]>")
