Progress System Tables Dan Foreman BravePoint, Inc. Email: danf@prodb.com BravePoint

Introduction - Dan Foreman Progress User since 1984 (V2.1) Presenter at lots of Progress conferences dating back to 1990 BravePoint

Introduction - Dan Foreman Author of: Progress Performance Tuning Guide Progress Database Admin Guide Progress Virtual System Tables Pro D&L - Dump/Load with short down time no matter how large the Database is ProMonitor – DB monitoring tool (uses VSTs) ProCheck – Monitor AppServers & WebSpeed BravePoint

What are Virtual System Tables Virtual System tables (VSTs) provide access to Database Shared Memory data that was previously only accessible through promon It’s possible to customize the user interface Accessible Locally or Remotely via the 4GL or SQL BravePoint

What are Virtual System Tables VSTs are Schema Tables They do not interfere with Application tables and Indexes VSTs are not part of the empty Database in V8 Enable with proutil <db> -C enablevst Are part of the V9/V10/V11 empty Database Are visible in the Dictionary (must view ‘Hidden’ tables) BravePoint

What are Virtual System Tables Some new tables and fields are not in the “native” empty DB unless: proutil –C updatevst is run Example: New VST fields added in V10.2B SP6 The feature must be enabled: Auditing: proutil enableauditing TDE Multi-Tenancy BravePoint

What are Virtual System Tables VSTs in the Standard empty DB V8.2 30 tables V8.3 35 tables V9.0-V9.1C 39 tables V9.1D 42 tables V10 45 tables V10.1B 47 tables BravePoint

What are Virtual System Tables promon can lag behind VSTs; Examples: _TableStat & _IndexStat (V8.3) _AreaStatus (V9 and later) Conversely VSTs can lag behind also: _Startup BravePoint

Resource Impact of VSTs There is no performance penalty for accessing VSTs with 2 exceptions that will be covered later VSTs take up no additional disk space except for the VST schema VSTs take up a small amount of memory to cache the additional schema Except –tablerangesize & -indexrangesize can increase DB Shared Memory by a surprising amount BravePoint

VST Implementation Whenever a record is requested, the table is checked to see if it’s a VST table. VST tables have table numbers between -16385 to -32768. The check is simple arithmetic which takes place regardless of whether VSTs have been enabled or not BravePoint

VST Implementation If a requested record is a VST record, the fields are populated from shared memory structures instead of from a database record BravePoint

VST Basics Each VST has one, unique index on the ‘Id’ field which is a ‘virtual’ RECID Each VST record is numbered starting at 1 and continues sequentially (some exceptions) It is not possible to add new indexes or triggers to VSTs BravePoint

VST Basics Some VST Record Counts are controlled by startup parameters _connect -n & -Mn _LockReq -n & -Mn _UserIO -n & -Mn _UserLock -n & -Mn _UserStatus -n & -Mn _Trans -n _Servers -Mn _Lock -L BravePoint

VST Basics The tables on the last slide are pre-populated at Database Broker startup time The empty slots have “?” values To avoid displaying the empty slots: FOR EACH _connect WHERE _connect-usr <> ? The <> operator is not a performance problem since VSTs are not really ‘indexed’ BravePoint

Which Database If a Client is connected to multiple Databases, the VST Statistics will be based upon the Current Working Database BravePoint

Updatable VSTs _Startup-spin -spin _Startup-APWQTime -pwqdelay _Startup-APWSTime -pwsdelay _Startup-APWBuffs -pwscan _Startup-APWMaxWrites -pwwmax _Startup- BravePoint

Updateable in V10.2B SP6 _Startup._Startup-LRU-Skips _Startup._Startup-NmsgWait _Startup._Startup-Pefetch-Delay _Startup._Startup-Prefetch-Factor _Startup._Startup-Prefetch-Num-Recs _Startup._Startup-Prefetch-Priority BravePoint

Updatable VSTs _MyConnect-NumSeqBuffers Added in V9 Same as –Bp parameter Number of Sequential Read Buffers Makes it possible to run a large report with a less severe impact on the -B Buffer Cache Example on the next slide BravePoint

Updatable VSTs DO FOR _MyConnect TRANSACTION : FIND FIRST _MyConnect. _myconnect-numseqbuffers = 32. END. RUN killer_report.p. _myconnect-numseqbuffers = 0. BravePoint

Updatable VSTs _StatBase._TableBase _StatBase._IndexBase V9 and later Used to set the window size of Tables and Indexes monitored using the _TableStat and _IndexStat VSTs or to reset (zero out) the statistics by moving the “window” up and then down BravePoint

Table & Index I/O VSTs Note that _TableStat and _IndexStat are available in V8.3 and very helpful for Storage Area Planning Use _IndexStat to identify unused indexes; COMPILE/XREF is only good for finding indexes that are defined but not used in the code; COMPILE/XREF is useless for Dynamic Queries BravePoint

VST Problems Potential Performance Killers _Block ( V8, useless in V9/V10/V11 ) _Lock ( FOR EACH _lock WHILE … ) _UserLock ( DO loops of 512 iterations ) Too Much Data for One Screen _Block-block x(1024) _UserLock-* Several arrays of 512 Missing Data _DBStatus BravePoint

VST Problems Storage VSTs are defined as INTEGERs which limits the maximum value to about 2 billion – prior to V10.1B Poorly documented (see my System Table Guide for better documentation) Stale Data _Lock (total # of records) compared to _DbStatus-NumLocks BravePoint

System Tables - MetaSchema _File _Field _Index _Index-Field _File-Trig _Field-Trig _Db _Sequence BravePoint

System Tables - Other SQL Auditing Encryption Multi-Tenancy Authentication (GRANTs & REVOKEs) Views Constraints Query Optimizer statistics Auditing Encryption Multi-Tenancy BravePoint

Usefulness VSTs are useful for two types of Progress users Application Developers Database Administrators Who else is there? BravePoint

For Developers Can view the lock status for an individual record _Lock VST shows all active locks and who has the lock We can easily find the Table name of the record being locked by doing a join to the _File table _Trans VST shows all active transactions BravePoint

For Developers Deadly Embrace (Deadlock) Detection _Connect “Delinquent” Active Transactions that grow the BI File to an abnormally large size _Trans BravePoint

For Developers Improve code efficiency by measuring: User DB Accesses, Reads, etc. Reads, Writes, Creates, Deletes by Table _TableStat Unused indexes; Logical Scatter Factor _IndexStat BravePoint

For Developers Name & attributes of a Field find _file “customer” no-lock. for each _field of _file no-lock : display _field-name _data-type _format _extent. end. BravePoint

DBAs - Tables of interest Monitoring the BI High Water Mark _DbStatus._dbstatus-bisize Available in V8.3 and later V8: BI Size in Bytes V9 & later: BI Size divided by BI Block Size BravePoint

DBAs - Tables of interest Monitoring Database Size promon doesn’t have the correct data Each Storage Area has a separate High Water Mark _AreaStatus _AreaStatus-AreaName _AreaStatus-HiWater _AreaStatus-TotBlocks _AreaStatus-LastExtent _AreaStatus-Extents BravePoint

DBAs - Tables of Interest Record Fragmentation _Record._Record-RecRead _Record._Record-FragRead dbanalys/tabanalys only shows how many Fragments exist, not how often the Fragments are accessed BravePoint

DBAs - Tables of Interest What Client is running what Program _Connect-Cache* Also Line# for 4GL Code Or SQL Statement Since V10.1C What Tables is a Client Hitting _UserTableStat Since V10.1B BravePoint

For More Information Progress System Tables Guide 200+ pages Covers V8 through V11 Covers VSTs and System Tables Has lots of undocumented information BravePoint

ProMonitor Written using (mostly) VSTs Source Code provided for most Programs Threshold Alerts warn of problems Monitor an unlimited number of DBs Monitors more than just VST Statistics Session Temp File I/O Client Memory Utilization DB Log File Monitoring DB Fragmentation/Scatter Monitoring AppServers, WebSpeed, AdminServer, etc BravePoint

Conclusion Questions? Dan Foreman danf@prodb.com www.BravePoint.com