Practical Advice and Guidance on the use of PeopleTools Performance Monitor David Kurtz Go-Faster Consultancy Ltd.

Practical PPM © Who Am I? Oracle Database Specialist –Independent consultant System Performance tuning –PeopleSoft ERP –Oracle RDBMS Book – UKOUG –PeopleSoft Technology SIG Committee

Practical PPM © Resources If you can’t hear me say so now. Please feel free to ask questions as we go along. The presentation is available from UKOUG Conference Website blog.psftdba.com

Practical PPM © Using PeopleSoft Performance Monitor Very Quick Overview Performance Tuning the Performance Monitor PPM Bugs & Fixes General Analyses –Events –Components –Network Latency Performance Trace Demo

Practical PPM © Think! The way you think about performance is more important than the tools you use. –Mostly, you need to focus on time. Further reading: –The Goal – Eli Goldratt –Optimising Oracle Performance - Millsap & Holt

Practical PPM © PPM – Key Features PeopleTools is used to Monitor PeopleTools –You don’t have to buy any more software –Monitoring system –v- Monitored systems Can and should use one monitor for many monitored Eg. Consolidated Portal/Application –Any version >= 8.44 can monitor any version When it works!

Practical PPM © Web Server Web Server Application Server (application logic) Application Server (application logic) APPQ PSAPPSRV DBMS (application data & meta-data DBMS (application data & meta-data SQL Tuxedo Message http / https Browser (presentation & JavaScript) Browser (presentation & JavaScript) Monitoring System PIA Servlet PSPPMSRV Monitored System Web Server (presentation logic) Web Server (presentation logic) PIA Servlet DBMS (application data & meta-data DBMS (application data & meta-data Tuxedo Message http / https Browser (presentation & JavaScript) Browser (presentation & JavaScript) Screen Paint Java Script Application Server (application logic) Application Server (application logic) APPQ PSAPPSRV PSMONITORSRV SQL PPMI Servlet 0345f1003x Monitor Servlet

Practical PPM © Performance Monitor Metrics Transactions –User activities in PIA that cause communications with application server –Sampled –Enabled to form a trace Events –Periodic samples –Usually initiated by monitoring agents –eg. CPU, Tuxedo counters

Practical PPM © Performance Monitor Transactions User activity in PIA Performance Monitoring Unit –Hierarchy of transactions Similar to Oracle event trace –recursive actions

Practical PPM © Transactions Stored to PSPMTRANSCURR table –As PMUs are closed moved to PSPMTRANSHIST –Later deleted or archived to PSPMTRANSARCH ERD downloadable from Metalink –And you will need to get to grips with it.

Practical PPM © ERD of Transaction PSTRANSHIST (C) PSPMSYSDEFN (B) PSPMAGENT (A) PSPMCONTEXTDEFN (C1) PSPMMETRICDEFN (M1) PSPMCONTEXTDEFN (C2) PSPMCONTEXTDEFN (C3) PSPMMETRICDEFN (M2) PSPMMETRICDEFN (M3) PSPMMETRICDEFN (M4) PSPMMETRICDEFN (M5) PSPMMETRICDEFN (M6) PSPMMETRICDEFN (M7) PM_CONTEXTID_n (n=1-3) PM_SYSTEMID PM_TRANS_DEFN_SET, PM_TRANS_DEFN_ID PM_AGENTID PM_METRICIDn (n=1-7) search criteria PSPMTRANSDEFN (E)

Practical PPM © Metrics Metric IDs specified on transaction definition PSPMTRANSDEFN –Metrics Types defined on PSPMMETRICDEFN Type 1: Counters (including timers) –Metric 4: Total Servlet Request time (ms) Type 2: Gauges –Metric 102: %CPU Used Type 3: Numeric Identifier –Metric 20: HTTP response code Type 4: String Identifier –Metric 27: File Name

Practical PPM © Transaction 101 Reported at entry and exit of PIA servlet –Context 1 Action=View Page –Context 2 IP Address= –Context 3 Session ID=AN7tpzSwpZc4kt9k8... –Additional Description psc/ps/EMPLOYEE/HRMS/c/UTILITIES.PTPERF_TEST.GBL

Practical PPM © Transaction metrics –Metric 19: Response Size (bytes) =17613 –Metric 20: Response Code =200 –Metric 22: Static Content Count =0 –Metric 23: Is this a Pagelet? =0

Practical PPM © Transaction Query Results PM_TOP_INST_ID PM_INSTANCE_ID PM_PARENT_INST_ID DBNAME PM_HOST_PORT PM_DOMAIN_NAME PM_AGENT_TYPE PM_INSTANCE PM_AGENT_STRT_DTTM PM_MON_STRT_DTTM OPRID PM_PERF_TRACE PM_PROCESS_ID PM_TRANS_DEFN_ID DESCR60 'CONTEXT1:'||C.PM_CONTEXTID_1||'-'||C1.PM_CONTEXT_LABEL||'='||C.PM_CONTEXT_VALUE … PM_TRANS_DURATION 'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1 … PM_ADDTNL_DESCR HR88 go-faster-3:7201:7202 ps WEBSERVER -1 16:12: :12: PS PS: :01: Reported at entry and exit of PIA servlet Context1:3-Session ID=AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt! ! Context2:2-IP Address= Context3:1-Action=View Page 1322 Metric1:Response Size (bytes)=17613 Metric2:Response Code=200 Metric3:Static Content Count=0 Metric4:Is this a Pagelet?=0 Metric5:=0 Metric6:=0 Metric7:=

Practical PPM © Events Do not have an explicit context –Collecting agent provide context Stored in PSPMEVENTHIST –Later deleted or archived to PSPMEVETARCH

Practical PPM © ERD of Events PSEVENTHIST (C) PSPMEVENTDEFN (E) PSPMSYSDEFN (B) PSPMAGENT (A) PSPMMETRICDEFN (M1) PSPMMETRICDEFN (M2) PSPMMETRICDEFN (M3) PSPMMETRICDEFN (M4) PSPMMETRICDEFN (M5) PSPMMETRICDEFN (M6) PSPMMETRICDEFN (M7) PM_AGENTIDPM_EVENT_DEFN_SET, PM_EVENT_DEFN_ID PM_SYSTEMIDsearch criteria PM_METRICIDn (n=1-7)

Practical PPM © Event Query Results DBNAME PM_HOST_PORT PM_AGENT_TYPE PM_DOMAIN_NAME PM_INSTANCE PM_AGENT_DTTM PM_INSTANCE_ID PM_EVENT_DEFN_ID DESCR60 'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1 'METRIC2:'||M2.PM_METRICLABEL||'='||C.PM_METRIC_VALUE2 'METRIC3:'||M3.PM_METRICLABEL||'='||C.PM_METRIC_VALUE3 'METRIC4:'||M4.PM_METRICLABEL||'='||C.PM_METRIC_VALUE4 'METRIC5:'||M5.PM_METRICLABEL||'='||C.PM_METRIC_VALUE5 'METRIC6:'||M6.PM_METRICLABEL||'='||C.PM_METRIC_VALUE6 'METRIC7:'||M7.PM_METRICLABEL||'='||C.PM_METRIC_VALUE7 PM_ADDTNL_DESCR HR88 go-faster-3:7201:7202 WEBSERVER ps -1 16:12: PSPING metrics fowarded from browser Metric1:Network Latency (ms)=435 Metric2:WebServer Latency (ms)=100 Metric3:AppServer Latency (ms)=561 Metric4:DB Latency (millisecs)=451 Metric5:=0 Metric6:=0 Metric7:IP Address= PS;AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt! !

Practical PPM © Tuning Performance Monitor Some of the delivered analytics do not perform well with even moderate data volumes –Set up the monitoring system to self monitor –Then you can generate PPM traces on the analytics –You will need additional indexes performance-monitor.html (YMMV) performance-monitor.html

Practical PPM © Purge Process Data normally held in history tables –PSPMTRANSHIST, PSPMEVENTHIST Clone tables –PSPMTRANSHISTCL, PSPMEVENTHISTCL PPM writes to tables specified in PSPMTABLEMAP –Archive process switches this to clone tables select * from pspmtablemap; PM_TRANS_TBL_NAME PM_EVENT_TBL_NAME PSPMTRANSHIST PSPMEVENTHIST

Practical PPM © PSPMTABLEMAP Archive/Purge switches PPM destination –Prevents concurrent INSERT and DELETE/Query operations –Saves read consistency problems on Oracle –Saves page locks on other databases PPM appears not to collect data during this processing –But it is written to clone tables –Archive process moves it to main hist tables after purge

Practical PPM © Purge Process Can’t Keep Up Platform generic query leads to full scan –Even if data has been deleted (manually) high water marks (HWM) on tables not reset Customisation –Oracle specific statement –May need to rebuild HIST tables to reset HWM In which case manually set PSPMTABLEMAP to clones, rebuild history tables, run archive process

Practical PPM © Performance Fix for Purge Vanilla Code &TransHistSQL.Open("SELECT … AND %DateTimeDiff(X.PM_MON_STRT_DTTM, %CurrentDateTimeIn) >= (PM_MAX_HIST_AGE * 24 * 60)"); Expands to … AND ROUND((CAST(( SYSDATE) AS DATE) - CAST((X.PM_MON_STRT_DTTM) AS DATE)) * 1440, 0) >= (PM_MAX_HIST_AGE * 24 * 60) My Suggestions … AND X.PM_MON_STRT_DTTM < SYSDATE - Z.PM_MAX_HIST_AGE See blog entry performance-monitor.html performance-monitor.html

Practical PPM © How much data? Control sampling –Proportion of transactions collected Depends upon activity on system On busy self-service system as little as 1 in 5000 –Event sampling frequency For each agent 5 minutes – 15 minutes Depends on whether you want to be able to see short- lived behaviours.

Practical PPM ©

Practical PPM © Recent problems in PT8.49 Prior to patch –Ports left open in close_wait –Unix systems run out of ports Get ‘Application Server is down’ errors –No Limit on Windows But the system does progressively slow down –POC applied to Tuxedo Connections capped at

Practical PPM © Outstanding problems in Tuxedo Queuing not reported –Events 300 and 301 Tuxedo Connections not reported –Event 300

Practical PPM © Practical Examples Simple Graphs of Events Cumulative Frequency Distributions Network Latency Performance Trace

Practical PPM © Simple Event Graphs You set an event collection interval All domains collect at that interval –But each has its own clock –Each collects at different times. If you have multiple web/app servers? –Need to aggregate for system wide view –Interpolate between points? | (PL/SQL package see notes for this slide)

Practical PPM © Simple Event Graphs Raw data from PSPMEVENTHIST or PSPMEVENTARCH –Extract into working storage tables Possibly two levels –Aggregating as you go

Practical PPM ©

Practical PPM ©

Practical PPM ©

Practical PPM ©

Practical PPM © Jolt Message Sizes Transaction 115 –Size of Jolt Messages into and out of Tuxedo –Message Written to disk If message larger than specfied size or would cause queue to become ¾ full –Default Queue Size is 64Kb Kernel Parameter (windows too) –Most systems need Kb

Practical PPM © Cumulative Frequency – ntile() SELECT pctile, MIN( ) FROM( SELECT NTILE(100) OVER (ORDER BY ) AS pctile FROM ) GROUP BY, pctile

Practical PPM ©

Practical PPM © Anatomy of a Transaction Simple PIA Transaction 101 – PIA entry/exit –115 – Jolt Message 400- Tuxedo Service –401 ICPanel –410 ICScript

Practical PPM © Anatomy of a Transaction Portal PIA Transaction –PMU is consolidated across databases. 106 – Portlet –115 – Jolt Message 400- Tuxedo Service –401 ICPanel –410 ICScript

Practical PPM ©

Practical PPM © Individual Components Now try the same analysis for specific components –Determine the top-n components by cumulative execution time PPM Analytic uses only event 401 –Doesn’t take web server or queuing time into account. I prefer to use event 101, 106 –But you have to join the transactions. –Component identification from transaction 401 contexts

Practical PPM © This area is Time spent in web server JVM Component that uploads an attachment

Practical PPM © Jolt Message to App Server increases – large attachments Component that uploads an attachment

Practical PPM © Time spent in ICPanel Service – possibly database or PeopleCode Interesting jump in response time Typical Component

Practical PPM © Time spent in ICPanel Service – possibly database or PeopleCode Significant jump in response time Custom PeopleCode Button

Practical PPM © Time spent in web server JVM Component that uploads attachment

Practical PPM © Network Latency Most transactions are sampled But three transactions are always recorded See PMTRANSDEFN.PM_SAMPLING_ENABLE –108: User Session logout, expiration, timeout, or error –109: User Session began (user logged in) –116: Redirected round trip time (network latency)

Practical PPM © Transaction 116 Network round trip from webserver to browser and back again –Includes network transmission time –Browser response time –Client IP address Although that could be load balancer or NAT –Operator ID LOCATION from HR database?

Practical PPM ©

Practical PPM © Analysis by Client IP IP addresses are of Routers not Clients Mon Jun 30 page 1 Login Durations by Client IP IP Address MIN AVG MED MAX VAR NUM_EVENTS , , , , , ,

Practical PPM © HR location of OPRID First three lines are different network topology Operator may not actually be at stated location *especially home workers Mon Jun 30 page 1 Login Durations by Location (min 10 logins) LOCATION CITY Cty MIN AVG MED MAX VAR EVENTS EXT-00-IN Bangalore IND , EXT-00-CZ Brno CZE , EXT-BK-UK Buckingham GBR , HOME Home Worker* GBR INT-L3 Liverpool GBR INT-BK Buckingham GBR INT-WS Walsall GBR

Practical PPM © After we fixed the problem…

Practical PPM © Analytics: Top Components

Practical PPM © Performance Trace Generates a group of PMUs for activity in a user session –Choose an ID to identify records later

Practical PPM © Performance Trace

Practical PPM © Performance Monitoring Unit Look at PMU Tree Demonstration

Questions?

Practical Advice and Guidance on the use of PeopleTools Performance Monitor David Kurtz Go-Faster Consultancy Ltd.