1. Efficient Coding and Indexing
Dan Foreman
BravePoint Inc.
BravePoint 2008

2. Introduction Ik kan niet spreken Nederlands maar Ik sprek goed Japans
BravePoint 2008

3. Introduction- Dan Foreman
Progress user since 1984
Guest Speaker at Progress Users Conference from and
Guest Speaker in Netherlands: 1993, ,
BravePoint 2008

4. Introduction- Dan Foreman
Author of:
Progress Performance Tuning Guide
Progress Database Admin Guide
Progress Virtual System Tables
V10 Database Administration Jumpstart
Online access for each Publication
ProMonitor - Performance Monitoring Tool
Pro D&L - Dump & Load with ultra-low down time
Progress Database Admin Resource Kit
Balanced Benchmark – benchmarking tool
BravePoint 2008

5. Who Are You? Progress V6, V7, V8, V9, V10
Production Database OS?: Unix, Linux, Windows
Largest Single Database
Largest Concurrent User Count
BravePoint 2008

6. Progress Access Methods
Indexes
Prefix B+ Tree
Does anyone know what that is?
Self Balancing
BravePoint 2008

7. Progress Access Methods
Recid/Rowid
Direct access to a block  record; no need to read an Index block
Very Fast (fewest I/Os)
But how often do you (or the application) know the Recid of a record?
BravePoint 2008

8. Record I/O
Reading one record may require reading several Database Blocks (Called “DB Requests” in promon)
1. Root Block of the B-Tree for that Index
2. One I/O for each B-Tree Level
3. If multi-index Query repeat #1 and #2
4. One I/O for each Record Fragment
BravePoint 2008

9. Indexing
The Current Record for most indexes on a table (that I have ever seen)
59 !!!
Cost of Indexing
Disk space, larger database size
Increased index rebuild time during D&L
Increased time to complete create, delete, and some updates
Higher Potential for Unused Indexes
BravePoint 2008

10. Indexing Rules Single Index Rules Multi-Index Rules
Beyond the Scope of this Presentation
Learn them!
Progress Database Design Guide
My Progress Performance Tuning Guide
BravePoint 2008

11. Indexing Quiz - The Easy Ones
for each cus where can-do(cus-typ,”a,b,c”)
for each cus where lookup(cus-typ,”a,b,c”) > 0
cus-typ is indexed
for each order where not back-ordered
back-ordered is indexed
for each xxx /* Index on [f1 + f2] */
where f1 GE v1a
and f1 LE v1b
and f2 GE v2a
and f2 LE v2b
BravePoint 2008

12. Indexing Quiz How do you know for sure which index is being used?
We can FORCE Progress with USE-INDEX but is that a good thing?
BravePoint 2008

13. USE-INDEX 100,000 Records Index on order# Index on cust# Query:
FOR EACH order
WHERE order.order# GE v-start#
AND order.order# LE v-end#
BY order.cust#
BravePoint 2008

14. USE-INDEX Query #1 Identifies a Bracket of 10,000 Orders
order# Index: 18 seconds
cust# Index: 62 seconds (3.4X slower)
No Surprise
Query #2 Identifies a Bracket of 90,000 Orders
order# Index: 191 seconds
cust# Index: 110 seconds (42% faster)
Big Surprise?
BravePoint 2008

15. Dangers of USE-INDEX
Progress can utilize multiple indexes in certain situations but USE-INDEX forces the use of one index even though the WHERE clause might reference fields from two or more indexes
If a new index is added to a table that is a better match than the one referenced with USE-INDEX, the potential benefits of the new index are not realized
“Fat fingers” when typing the index name could force Progress to use the wrong index
BravePoint 2008

16. Indexing Quiz COMPILE code.p XREF code.xrf x.p x.p 1 COMPILE x.p ...
x.p x.p 1 SEARCH sports.x f1f2f3asc WHOLE-INDEX
x.p x.p 1 SORT-ACCESS sports.x f1
x.p x.p 1 SORT-ACCESS sports.x f2
x.p x.p 1 SORT-ACCESS sports.x f3
x.p x.p 4 SEARCH sports.x f1f2f3asc WHOLE-INDEX
BravePoint 2008

17. COMPILE XREF
Useless for Dynamic Queries
BravePoint 2008

18. Indexing Quiz
Now we know what index or indexes were chosen by the compiler
Are we sure that those are the same Indexes used at run-time?
How good were our brackets?
BravePoint 2008

19. Indexing Quiz -zqil (Zecret Query Info Log) Client Startup Option
Writes compile and execution information for a query to the database .lg file each time the query is executed
BravePoint 2008

20. Indexing Quiz -zqil output
FOR EACH customer WHERE name EQ "John" AND Comments CONTAINS "hello":
==Compiled Query Resolution Method: Query No. 1== (6135)
11:02:43 AND (6137)
11:02:43 INDEX (6157)
11:02:43 INDEX EQUALITY (6154)
11:02:43
==Server Query execution Method Query No. 1== (6136)
11:02:43 QUERY-BY-WORD (6164)
11:02: INDEX 13 CONTAINS HELLO (6148)
11:02:43 INDEX 15 (6141)
BravePoint 2008

21. Indexing Quiz Index A for each table How far does the bracket extend?
F1 + F2 + F3 + F4
for each table
where f1 EQ v1
and ( f2 EQ v2a OR f2 EQ v2b )
and f3 EQ v3
How far does the bracket extend?
Can you prove how far the bracket extends?
BravePoint 2008

22. Indexing Quiz Can we get better Bracketing? for each x where f1 EQ v1
and ( f2 EQ v2a OR f2 EQ v2b )
and f3 EQ v3
-zqil shows:
Compiled Query Resolution Method: Query No.1 (6135)
INDEX EQUALITY (6157)
Can we get better Bracketing?
BravePoint 2008

23. Indexing Quiz for each x where
(f1 EQ v1 and f2 EQ v2a and f3 EQ v3 ) OR
(f1 EQ v1 and f2 EQ v2b and f3 EQ v3 ):
-zqil shows:
==Compiled Query Resolution Method: Query No. 1== (6135)
14:07:17 OR (6138)
14:07:17 INDEX EQUALITY (6154)
14:07:17
==Server Query execution Method Query No. 1== (6136)
14:07:17 DUPLICATE-REMOVE (6163)
14:07:17 OR (6138)
14:07: INDEX 36 (6141)
BravePoint 2008

24. Indexing Quiz EACH customer WHERE sales-rep = “bbb”
AND comments CONTAINS “Prog*”
COMPILE/XREF shows 2 indexes
x.p x.p 1 SEARCH Customer Comments
x.p x.p 1 SEARCH Customer Sales-Rep
BravePoint 2008

25. Indexing Quiz
-zqil shows only one index is actually used at run time; Why?
==Compiled Query Resolution Method: Query No. 1= (6135)
17:32:44 AND (6137)
17:32:44 INDEX (6157)
17:32:44 INDEX EQUALITY (6154)
==Server Query execution Method Query No. 1== (6136)
17:32:45 DUPLICATE-REMOVE (6163)
17:32:45 QUERY-BY-WORD (6164)
17:32: INDEX 13 CONTAINS Prog* (6148)
Full info on -zqil in Progress Performance Tuning Guide
BravePoint 2008

26. Client Log Manager Added in V9.1D SP01
Not just for maintaining AS/WS logs
Startup Parameter
Method/Attribute
Description
-clientlog
LOGFILE-NAME
Name of log
-logentrytypes
LOG-ENTRY-TYPES
Type of log
-logginglevel
LOGGING-LEVEL
How verbose
-numlogfiles
NUM-LOG-FILES
# of logs to keep
-logthreshold
LOG-THRESHOLD
Size threshold
BravePoint 2009

27. Client Log Manager Output
<run: for each customer where creditlimit GT 10000> QRYINFO Query Plan: C:\PROTMP\v10\101c\p92032_Untitled1.ped line 1 QRYINFO QueryId: 0x13d67f4 QRYINFO Type: FOR Statement QRYINFO Client Sort: N QRYINFO Scrolling: N QRYINFO Table: s2k.Customer QRYINFO Indexes: CustNum QRYINFO Query Statistics: C:\PROTMP\v10\101c\p92032_Untitled1.ped line 1 QRYINFO DB Blocks accessed: QRYINFO s2k : 2246 QRYINFO DB Reads: QRYINFO Table: s2k.Customer : 1117 QRYINFO Index: Customer.CustNum : 1118 QRYINFO s2k.Customer Table: QRYINFO 4GL Records: 909 QRYINFO Records from server: 909 QRYINFO Useful: 909 QRYINFO Failed: 0 QRYINFO Select By Client: N
BravePoint 2009

28. Indexing Quiz for each table Index on f1
where f1 = v1
or f1 = v2
or f1 = v3
Index on f1
Can this code be made more flexible?
Remember that
for each x where lookup(f1,”a,b,c”) > 0
can’t Bracket on an Index
BravePoint 2008

29. Indexing Quiz vstr = v1 + ‘|’ + v2 + ‘|’ + v3.
for each x where f1 contains vstr
Flexible (can have one or more values)
Very Fast
Word Indexing Cost is Zero (for this code)
The size of the vstr variable can’t exceed 180 bytes until V10.1B
Is there another way?
BravePoint 2008

30. Indexing Quiz Temp Table Join
/* a browser let’s the user select values and store in temp table*/
for each tmp,
each cust where cust.f1 = tmp.f1
Alternatively use a Master File instead of a Temp-Table
BravePoint 2008

31. Indexing Quiz Which is better (faster)?
Multiple Small (i.e. Single Field) Indexes
Few Larger Multi-Field Indexes
Answer: There is no one correct answer
BravePoint 2008

32. Indexing Quiz What is the fastest way to find unused Indexes?
Indexes defined but never used in the code; beware of Dynamic Queries
Indexes that are defined and used in the code but never accessed because:
The code is never executed
The query is never executed because of IF logic
BravePoint 2008

33. Indexing Quiz _IndexStat Virtual System Table
Database Startup options are required to make data in this VST table ‘visible’
V8: -indexbase, -indexlimit
V9/V10: -baseindex, -indexrangesize
BravePoint 2008

34. Efficient Coding
BravePoint 2008

35. Transactions FOR EACH trx EXCLUSIVE : DELETE trx. END.
Simple and easy to code
But very slow
One Transaction start & stop per Record
BravePoint 2008

36. Transactions REPEAT TRANSACTION :
FOR EACH trx EXCLUSIVE cnt = 1 to 500 :
DELETE trx.
END.
IF NOT CAN-FIND ( FIRST trx ) THEN LEAVE.
One Transaction start & stop per 500 Records
Larger counter = bigger BI & more record locks
Of course you can add a WHERE phrase
BravePoint 2008

37. Transactions Apply the same method to loading records main-blk:
REPEAT TRANSACTION :
REPEAT cnt = 1 TO 500
ON ENDKEY UNDO, LEAVE main-blk:
CREATE trx.
IMPORT trx.
END.
BravePoint 2008

38. Optional EACH FOR EACH orderline, item OF orderline :
Assumes an item Record MUST Exist for each line; error if not matching record
25% Faster than: EACH item OF orderline
Helps document the one-to-one relationship
OPEN QUERY requires a EACH, FIRST, or LAST
Use “FIRST item” for a 12-14% gain
BravePoint 2008

39. TEMP-TABLEs In V9/V10 use EMPTY TEMP-TABLE
Unfortunately if the DEFINE TEMP-TABLE uses NO-UNDO, the EMPTY TEMP-TABLE Statement cannot be used inside of a TRANSACTION
BravePoint 2008

40. TEMP-TABLEs
Starting in V9 TEMP-TABLE performance decreased (12-20% in benchmarks)
Reason: the DBI file Block size was changed to 8k and the Records per Block to 128
The change was reversed in V9.1C05
There are many other version dependent changes to TEMP-TABLE performance
BravePoint 2008

41. WORKFILE &WORK-TABLE 100% Memory Resident
No Indexes; FIND FIRST followed by FIND LAST requires a record by record scan
If the record size is increased enough a new ‘record’ needs to be added pushing all the subsequent records down
Did you know that you can potentially crash a SYSTEM with 3 lines of code; do you know what those statements are?
BravePoint 2008

42. Record Copying
For copying data between records from Fastest to Slowest
RAW-TRANSFER (5.2X Faster than BUFFER-COPY but has some restrictions like unique key collisions when used in the same table)
BUFFER-COPY (34% Faster than ASSIGN)
ASSIGN the individual fields (slowest)
BravePoint 2008

43. Other Coding Techniques
CAN-DO/LOOKUP/INDEX versus ORs
But don’t use in a WHERE Clause with an indexed field
Prior to V10 there were big differences; now there are none
Array Initialization: array-name = “”
3.5X slower for a 12 element CHAR Array
383X slower for a 365 element INT Array
BravePoint 2008

44. Other Coding Techniques
ASSIGN for Multiple Assignments
a = b.
c = d.
ASSIGN
a = b
BravePoint 2008

45. Other Coding Techniques
SHARED BUFFER or BUFFER PARAMETER versus passing RECID/ROWID
NO-LOCK - especially with Remote Clients (-H/-S)
NO-UNDO
DEF VARIABLE
DEF TEMP-TABLE/WORKFILE
DEF PARAMETER
BravePoint 2008

46. Startup Options Quick Request (-q)
Especially if the code resides on a different computer
Not for developers
Not so useful with Procedure Libraries
BravePoint 2008

47. Startup Options Progress Packet Size (-Mm)
For Remote Client (-H, -S) Connections
Will make NO-LOCK Reads much faster
Will make EXCLUSIVE-LOCK and SHARE-LOCK Reads a little slower
Must be used on the Client and the Database which means ALL DBs that are commonly connected to
Ignored on SQL 92 Connections
BravePoint 2008

48. Good Ways to Find Slow Code
Use -yx to find slow programs in a session
Use this to locate which program is slow
Use the Undocumented Application Profiler
Use this option to find what statements are slow
-zprofile in V8
-profile in V9/V10
Old: $DLC/src/samples/profiler/readme.doc
New: Download from Progress web site
BravePoint 2008

49. Conclusion
Vragen?
BravePoint 2008

50. If you need further assistance:
Dan Foreman
Dank U wel!
BravePoint 2008