1 Michigan Progress Users Group 2015 Strength. Strategy. Stability. Identifying Performance Issues in Progress Code and Queries

2 Michigan Progress Users Group 2015 Introduction- Dan Foreman Progress since 1984 (V2.1) Guest speaker at dozens of Progress conferences dating back to 1990 Employee of Progress Software since October 2014 when Progress acquired BravePoint

3 Michigan Progress Users Group 2015 Introduction- Dan Foreman Author of: –Progress Performance Tuning Guide –Progress Database Admin Guide –Progress System Tables Guide –A variety of Whitepapers –Available in paper or PDF I am a Michigander ! (Byron Center > Grandville) Interests include: –Japan (speak reasonably good Japanese) –Trikes (??) –Basketball 3

4 Michigan Progress Users Group 2015 Just got back from Japan

5 Michigan Progress Users Group 2015

6

7 Warning! Graphic content may be unsettling for some viewers

8 Michigan Progress Users Group 2015

9 Disclaimers I use the term 4GL not ABL I can’t see your faces so I won’t have a clue about if you’re bored, etc. Please let the local moderator know if you can’t see/hear 9

10 Michigan Progress Users Group 2015 Client Monitoring - The –y* Family -y (V4) –yc –yx (V6) -yd (V7) Interesting Data (-yc –yd –y) –-mmax High Water Mark –All Client Startup Options (including defaults) &.pf files –RCD Temp File I/O -yx stuff –All code executed by Client (including database triggers) –Individual program execution time –Number of times a program was run 10

11 Michigan Progress Users Group 2015 Client Monitoring - The –y* Family -pf c:\DLC\11.5.0\startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d mdy,-numsep 44, -numdec 46,(end.pf),-db c:/pm/db/pm, -yd,-yx,-p pm-menu.p … Program access statistics: Times Bytes Reads from temp file: 0 0 Writes to temp file: 0 0 Loads of.r programs: 0 0 Saves of compilation.r's: 0 0 Compilations of.p's: Checks of files with stat: 54 0 Memory usage summary: Current Max Used Limit Stack usage (-s): Local buffer usage: R-code Execution Buffer:

12 Michigan Progress Users Group 2015 Client Temporary Files PrefixParameterDescriptionSize SRT-TB & -TMSort file. Used for non-indexed sorting.100mb DBI-Bt & -tmpbsizeTEMP-TABLE database.500mb RCD-mmaxR-Code. New in V10. Swap area for –mmax execution buffer. 100mb LBINoneLocal Before Image. WORKFILEs and logging for variables, etc. not defined with NO-UNDO. Varies On *ix not visible unless –t Client parameter is used -T Client parameter specifics location Resource consumption is per Client Size is a threshold when I would pay closer attention to the source of the activity

13 Michigan Progress Users Group 2015 Application Profiler – Ancient History Added by Tim Sargent in 1999 Initial Version - Progress V8.2A Initially used a zecret startup option: -zprofile Use –profile for V9 and later 13

14 Michigan Progress Users Group 2015 Application Profiler – Recent History Renamed Profiler Control Tool in 2003 Big Refresh in V11.6 The Application Profiler (AP) is not formally supported by PSC: The Profiler Control Tool is freeware and is not supported by Progress Technical Support. It is provided as is, has not undergone a quality assurance process, and is not maintained by Progress engineering. Please do not contact Progress Technical Support regarding any aspect of this tool or its use. But I have never heard of anyone having issues with it 14

15 Michigan Progress Users Group 2015 AP Basics Old Location: $DLC/src/samples/profiler Current Location: Progress Communities & Knowledgebase Excellent documentation (readme.doc) by the author Requires a GUI (prowin[32].exe) license to use the supplied code But the code being profiled doesn’t need to be GUI code 15

16 Michigan Progress Users Group 2015 AP Updates The Profiler Control Tool is available on Progress Communities: al/w/openedgegeneral/1980.profiler-control-tool

17 Michigan Progress Users Group 2015 What Is Application Profiling? From Tim Sargent’s original documentation “A performance profiler is a common and useful tool for a software developer. In brief, a profiler provides a “profile” of a particular execution. A profiler generally provides timing information and call-tree information; with that, an engineer can analyze where their program is spending most of its time and what part of the application is calling what other part of the application.” 17

18 Michigan Progress Users Group 2015 AP Basics Two Main Parts –Generating the Profiling Data Requires a separate DB for the data –Viewing & Analyzing the Profiling Data 18

19 Michigan Progress Users Group 2015 AP Getting Started Client Startup Option: –-profile Minimum entry in the Config file is: –-filename There is also a PROFILER System Handle in the 4GL that can be used to specify the equivalents of the command line options 19

20 Michigan Progress Users Group 2015 Other AP Options -listings will attempt to create COMPILE DEBUG-LISTINGs for the code being Profiled IF the source code is available in PROPATH This is a highly recommended option But if the PROPATH for the Client (probably using R-code) being profiled is different than the PROPATH for the source…watch out 20

21 Michigan Progress Users Group 2015 AP Raw Data 1 05/06/2012 "Generic" 13:47:22 "Dan". 10 "promon-b.p" "c:\pm\profiler\src\dbg_promon-b_09a01752" "promon-a.p" "c:\pm\profiler\src\dbg_promon-a_07a01752" "promon2.p" "c:\pm\profiler\src\dbg_promon2_02a01752" "promon-k.p" "c:\pm\profiler\src\dbg_promon-k_04a01752" "promon-n.p" "c:\pm\profiler\src\dbg_promon-n_0ba01752" "val-key.p" "c:\pm\profiler\src\dbg_val-key_01a01752" "promon1.p" "c:\pm\profiler\src\dbg_promon1_00a01752" "promon-g.p" "c:\pm\profiler\src\dbg_promon-g_03a01752" "promon-c1.p" "c:\pm\profiler\src\dbg_promon-c1_05a01752" … 21

22 Michigan Progress Users Group 2015 Samples Load the Profiling Data 22

23 Michigan Progress Users Group 2015 Samples Session Data 23

24 Michigan Progress Users Group

25 Michigan Progress Users Group 2015 Other Stuff AP Data doesn’t include “User Think Time” 25

26 Michigan Progress Users Group 2015 Profiler Control Tool (2003) Adds support for Dynamic Objects No separate DB required

27 Michigan Progress Users Group 2015 Profiler Viewer Tool Only for viewing of raw profiler data Written in Visual Basic Requires Progress a minimum of V10.2A Not friendly on all Windows platforms (Win 7 64-bit)

28 Michigan Progress Users Group 2015 Other Diagnostic Tools – Virtual System Tables _UserTableStat –Records CRUD Table I/O by Client –Find out what Database I/O is being performed by a specific Client –More granular than _TableStat (although _TableStat has DB Reads info) –Need –tablerangesize on DB startup –V10.1B and later 28

29 Michigan Progress Users Group 2015 Other Diagnostic Tools – Virtual System Tables _UserTableStat –Won’t identify TEMP-TABLE I/O –Refer to my “Temp-Table Tuning” presentation for new V11 features that allow TEMP-TABLE monitoring conferences/w/exchange/2349.emea-pug-challenge breakout-sessionshttps://community.progress.com/community_groups/user_ conferences/w/exchange/2349.emea-pug-challenge breakout-sessions

30 Michigan Progress Users Group 2015 Other Diagnostic Tools – Virtual System Tables _UserIndexStat –Similar to _UserTableStat but for indexes –Need –indexrangesize on DB startup –An Excellent way to find unused indexes Warning: A large –n (Max # of DB connections) plus a large –indexrangesize / -tablerangesize can increase the amount of DB shared memory allocated at DB startup by a surprising amount; test first 30

31 Michigan Progress Users Group 2015 Other Diagnostic Tools – Virtual System Tables Other useful VSTs –_Lock Lock Table Info Dangerous resource consumption in certain environments; test! Semi-Workaround: for each _lock while _lock-recid <> ? : Performance Fixed in V11.4 –_Trans Transaction Duration

32 Michigan Progress Users Group 2015 Statement Caching Strange name for the feature that permits querying the Program Stack for any 4GL Client New in V10.1C+ Also shows SQL Statements for SQL Clients Enabled & disabled via promon only Data is visible via: –promon –_Connect Virtual System Table (VST) Data is only visible if Client ‘touches’ the DB 32

33 Michigan Progress Users Group 2015 Statement Caching Sample Output 33 Client Detail DB: amos Date: 10/04/12 Time: 10:19 Cache Timestamp: Thu Oct 4 10:19: Line# Program Stack 2860 queue/amp-q01.pp BUILD-RAWQUEUE..\..\amos\obj\p\amp-q01.r DISP-UPD-PROCEDURE..\..\amos\obj\p\amp-q01.r 9474 DISPLAY-FRAMEDATA..\..\amos\obj\p\amp-q01.r 8911 UPD-REC1..\..\amos\obj\p\amp-q01.r 7577 UPD-REC..\..\amos\obj\p\amp-q01.r \..\amos\obj\p\amp-q01.r 5602 RUN-PROGRAM p/amp-menu.p

34 Michigan Progress Users Group 2015 Statement Caching – Warning If the SC data doesn’t fit in shared memory it’s “swapped” to a disk file with a.cst suffix The location of these files is defined in promon I/O can be very high if the Progress clients have very dynamic stacks 34

35 Michigan Progress Users Group 2015 Other Warnings - Bugs DB Crashing Bug fixed in V10.2A SP3.cst files permissions bug – workaround by using a specified directory that is writeable by all Newly found (at my customer) in V10.2B SP6, fixed in a hotfix (I was told) September 2015 had an issue with V10.2B SP 08 Hotfix 08; didn’t shut down the DB but held some latches that eventually forced us to shutdown the DB 35

36 Michigan Progress Users Group 2015 Other Diagnostic Tools Client Log Manager Basic Client Startup Options: –-clientlogquery.txt –-logentrytypesQryInfo –-logginglevel3 In the 4GL use the LOG-MANAGER System Handle 36

37 Michigan Progress Users Group 2015 Client Log Sample QRYINFO Query Plan: C:\protmp\tmp\p91026_Untitled1.ped line 1 QRYINFO QueryId: 0x2cdb8e0 QRYINFO Type: FOR Statement QRYINFO Client Sort: N QRYINFO Scrolling: N QRYINFO Table:../s2k.Customer QRYINFO Indexes: CustNum QRYINFO Query Statistics: C:\protmp\tmp\p91026_Untitled1.ped line 1 QRYINFO QueryId: 0x2cdb8e0 QRYINFO DB Blocks accessed: QRYINFO../s2k : 2310 QRYINFO DB Reads: QRYINFO Table:../s2k.Customer : 1117 QRYINFO Index: Customer.CustNum : 1118 QRYINFO../s2k.Customer Table: QRYINFO 4GL Records: 676 QRYINFO Records from server: 676 QRYINFO Useful: 676 QRYINFO Failed: 0 QRYINFO Select By Client: N 37

38 Michigan Progress Users Group 2015 Other Diagnostic Tools -zqil Zecret Query Info Log 4GL Client Startup Option Output written to Database.lg file Shows: –Bracketing –Index Selection versus Usage Appears to be broken in V11…unsure of the extent of the damage 38

39 Michigan Progress Users Group 2015 Example #1A One Index with 3 fields (f1+f2+f3) for each table where f1 = v1 andf2 = v2 andf3 = v3 Abbreviated -zqil Output ==Compiled Query Resolution Method: Query No.1==(6135) 21:21:24 INDEX EQUALITY (6157) 39

40 Michigan Progress Users Group 2015 Example #1B One Index with 3 fields (f1+f2+f3) for each table where f1 = v1 andf2 <> v2 andf3 = v3 -zqil Output ==Compiled Query Resolution Method: Query No. 1==(6135) 21:21:24 INDEX EQUALITY (6157) 40

41 Michigan Progress Users Group 2015 Example #2 Example disproving COMPILE XREF WHERE name = "dan" AND comments CONTAINS "hold*" ==Compiled Query Resolution Method: Query No. 2== AND INDEX INDEX EQUALITY ==Server Query execution Method Query No. 2== DUPLICATE-REMOVE QUERY-BY-WORD INDEX 13 CONTAINS HOLD* 41

42 Michigan Progress Users Group 2015 Conclusion Questions? Thank you for attending! Dan Foreman – 42