PeopleTools Application Server & BEA Tuxedo Internals david.kurtz@go-faster.co.uk Go-Faster Consultancy Ltd. http://www.go-faster.co.uk

Who am I? Independent Consultant Performance Tuning PeopleSoft / Oracle

Rules of Engagement Interrupt me: if you think I have got something wrong if you have a question if you can’t hear me I put this slide in to make sure I say this

BEA Tuxedo Historical Overview Tuxedo Internal Architecture PeopleSoft 3-tier client implementation How does PeopleSoft Internet Architecture (PIA) fit in Configuration issues Monitoring

Brief Historical Background In the beginning monolithic Presentation Business Logic Data Access (DBMS) IBM Mainframe

Client Server Client Server LAN Two-tier client-server SQL Presentation Business Logic Data Access (DBMS) SQL Raw Data

3-Tier Client Server Client Server LAN “1st Generation” GUI + Application Logic Data Access (DBMS) Business Logic Client Server LAN Request Response

3-Tier Client Server “2nd Generation” - Tuxedo Business Logic Data Management TUXEDO Servers Presentation Logic TUXEDO Clients DBMS 1 ... DBMS n Database Servers Service 1 Service 2 Service n TUXEDO Messaging SQL

Where did Tuxedo come from? Bell Labs - 1993 Transactions for UniX Extended for Distributed Operations The key words here are distributed operations. Different parts of a transaction occur on different nodes

What is Tuxedo? It is the knots on the ends of the string! Imagine a telephone make of string and tin cans. Every child has made one If a tuxedo application where that telephone, the tuxedo bits would be the knots on the end of the string

PeopleSoft 2-Tier Client Presentation Logic Panel Load Save Field Change etc... PeopleSoft windows client is written in C Imagine cutting up the two tier client

PeopleSoft 3-Tier Client Presentation Logic Tuxedo Client Tuxedo Server Panel Load Save Field Change etc... Client Side Server Side

PeopleSoft 2-Tier Client Client Connects Directly to the Database 1 database connection per window inactivity timeout DATABASE

PeopleSoft 2-Tier Client Client Connects Directly to the Database 1 database connection per window inactivity timeout What happens in 3-Tier? DATABASE

PS/Tuxedo Internal Structure 1st Application Server Process Started Bulletin Board Liaison (BBL) Reads PSTUXCFG configuration file Shared Memory Segment Bulletin Board (BB) BB DATABASE PSTUXCFG BBL

PS/Tuxedo Internal Structure Workstation Listener (WSL) Listens for connection request from clients on a specified port Client initially contacts the WSL WSL BB DATABASE BBL

PS/Tuxedo Internal Structure Workstation Listener (WSL) Client needs to know where to find the WSL Machine and port specified in Configuration Manager WSL BB BBL

PS/Tuxedo Internal Structure Workstation Handler (WSH) handles traffic from client after connection Client is assigned to a port on the WSH WSH port range can be constrained to match a firewall Additional WSH can be spawned if necessary Compress Messages WSL WSH BB DATABASE BBL

PS/Tuxedo Internal Structure PSAUTH AUTHQ PSAPPSRV APPQ PSQCKSRV QCKQ PSQRYSRV QRYQ PSAPISRV APIQ PSSAMSRV SAMQ WSL WSH BB DATABASE BBL

How does it work? BB PSAUTH AUTHQ PSAPPSRV APPQ PSQCKSRV QCKQ PSQRYSRV QRYQ PSAPISRV APIQ WSHQ PSSAMSRV SAMQ WSL WSH BB DATABASE BBL

How does PIA work? Java Client & PIA use Jolt Java Class Library Jolt Listener maps Java functions to Tuxedo Messages BEA Jolt connects Java clients to applications built using the BEA TUXEDO system. The TUXEDO system provides a set of modular services, each offering specific functionality related to the application as a whole. For example, a simple banking application might have services such as INQUIRY, WITHDRAW, TRANSFER, and DEPOSIT. In PeopleSoft it is the functionality of the client Typically, service requests are implemented in C or COBOL as a sequence of calls to a program library. The Jolt Server implementation acts as a proxy for the Jolt client, invoking the TUXEDO service on behalf of the client. The BEA Jolt 1.2 Server accepts requests from the Jolt clients and maps those requests into TUXEDO service requests. Diagram is taken from BEA Jolt Manual

Jolt Internal Structure PSAUTH AUTHQ PSAPPSRV APPQ PSQCKSRV QCKQ PSQRYSRV QRYQ JSHQ PSAPISRV APIQ JSL JSH JREPSRV PSSAMSRV SAMQ WSL WSH BB JREPOSITORY DATABASE BBL

New structure for PIA (Apache Webserver) Sessionid = “xy1k…vm8010 http 1.1 address+cookie (stateless) Cookie Java JVM Servlet Engine Servlet thread IS stateful Servlet Container Servlet Webserver Servlet Thread Servlet Thread HTTPD Client Servlet Thread Client Disk (static objects) mod_jserv Port 8010 Tuxedo Application Server JSL JSH JSH JSH

What is the difference between the web servers? Apache Separate Servlet Engine Separate encryption module (Raven) Weblogic Contains an integral servlet container

Progressive Connection Concentration 35000 users Typical picture 35,000 users 1000 concurrent connections 1000 users active in the last n minutes where n is the Java + JSH timeout therefore need 1000 threads but maybe only 4 JVMs 100 JSH assumes 10 clients per JSH 10-20 PSAPPSRV processes each PSAPPSRV connects to database 1 database 1000 connections 4 JVMs 1000 Java Threads 100 JSHs 10-20 PSAPPSRVs 1 database

Configuration psappsrv.val psadmin (configuration dialogue) psappsrv.cfg ubbgen (PS Utility) psappsrv.ubx psappsrv.ubb tmloadcf (tuxedo process) PSTUXCFG BBL (domain startup)

Configuration Files psappsrv.cfg psappsrv.val psappsrv.ubx contains configurable parameters psappsrv.val validation file for configurable values psappsrv.ubx configuration template for Tuxedo domain psappsrv.ubb configuration file for Tuxedo domain PSTUXCFG compiled configuration file read by BBL at domain startup

psappsrv.ubx -v- psappsrv.ubb psappsrv.ubx & psappsrv.cfg PeopleSoft file make changes to these files difficult to read psappsrv.ubb Tuxedo file never make changes to this file easier to read this file is compiled by tmloadcf to generate domain configuration file PSTUXCFG variables fully resolved

psappsrv.ubb Five (or Six) Sections Resources Machine Groups Servers resource limits -> BB sizing Machine Resource limits. Environment (Network Connection between Machines) Groups Logical Split (BASE, APPSRV, JREPGRP, JSLGRP) Servers Number of servers, Command line Services Timeout, Load, Priority

Application Server Configuration Issue Move quick PSAPPSRV services into a second server (PSQCKSRV) (y/n)? <=PT7.58 this question is misleading services are additionally advertised on PSQCKSRV as well as PSAPPSRV so they are advertised on more than one queue >=PT7.59 and PT8.1 services are de-advertised from PSAPPSRV quick services not used in PIA, only IC* services so not a problem

PeopleTools <=7.58 Quick Service BB PSAPPSRV APPQ PSQCKSRV QCKQ WSL WSH BB DATABASE BBL

Inside the Bulletin Board Without Load Balancing (default) Table of servers process where each service is advertised Read and populated as a stack WSL looks for first free server where request can be enqueued This can cause problems PSAPPSRV PSQCKSRV PSQCKSRV PSQCKSRV PSAPPSRV PSAPPSRV PSAPPSRV

Load Balancing Load Balancing is about choosing the queue upon which to place the message by default all requests have the same load 50 PSAPPSRV APPQ 50 PSQCKSRV QCKQ

Load Balancing Trace Add -r to application server command line CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S PSAPPSRV” -r enables trace to file ‘stderr’ (not channel 2) -e overrides the output filename process trace file with Tuxedo utility ‘txrpt’ obtain average service time make load proportional to average service time

Load Balancing Trace Output from TXRPT SVCNAME 18p-19p TOTALS Num/Avg Num/Avg --------------- -------- ------- SqlRequest 16/0.10 16/0.10 MgrClear 5/0.04 5/0.04 RamList 1/0.43 1/0.43 SamGetParmsSvc 1/0.50 1/0.50 --------------- ------- ------- TOTALS 23/0.12 23/0.12

Load Balancing load = execution time (seconds) x 100 78 386 135 599 50 PSAPPSRV APPQ 599 50 63 3 7 6 30 PSQCKSRV QCKQ 116

Multiple Queues on Busy System You can have many server processes on a single queue BEA recommend no more that 10 server processes per queue If you have queue contention it is likely to impede the WSH/JSH processes enqueuing requests. You are unlikely to observe queuing with PQ. So, if you want more than 10 PSAPPSRV processes you should use two APPQ queues In, which case you should to enable load balancing and you must calculate the load. Less significant with PIA because services different.

Multiple Queues on Busy System PSAPPSRV APPQ1 PSAPPSRV APPQ2 WSL WSH BB DATABASE BBL

Multiple Queues on Busy System psadmin will not correctly calculate MAXACCESSERS MAXSERVICES tmadmin->bbs will report maximum number of services count of all services advertised on each server process and if you have multiple work station listeners MAXWSCLIENTS

Service Priority Priority is about queue jumping servers de-queue the request 9 out of 10 de-queued on the basis of priority 1 out 10 is taken from the front of the queue 80 50 PSQCKSRV QCKQ

Service Priority All PeopleSoft Services have same priority specified in SERVICES section in psappsrv.ubb therefore there is no queue jumping no reason to believe that it would be advantageous

Tuning Tip Financials On-line journal edit & post - remote call Service initiates a Cobol process Process can take minutes RemoteCall services can block APPQ Move RemoteCall service to a separate queue

Multi-Node Domain 1 domain on many nodes BB BB DBBL WSL WSH BBL PSAPPSRV APPQ BRIDGE TLISTEN BRIDGE WSL WSH BBL BB PSAPPSRV APPQ DATABASE TLISTEN

Multi-Node Domain Distinct Bulletin Board Liaison process Master Node TLISTEN process Listens for incoming requests BRIDGE Sends & Receives messages between nodes in a load balancing environment configurable bridge load add to remote queues limiting number of messages sent across the bridge

Multi-Node Domain DO ATTEMPT THIS WITHOUT BEA CONSULTANCY At UBS PT 7.5x susceptible to network errors integrated into HP service guard ultimately this was removed from production because of transient network errors between the nodes causing the domain to ‘partition’

So what is going on? tmadmin Tuxedo command line interface also available within psadmin

tmadmin commands pq - printqueues amount of work queued and waiting not always accurate

tmadmin commands psr - printservers reports activity of individual server processes

tmadmin commands psc - printservices each service on each server

tmadmin commands Lots of other tmadmin commands documented in the Tuxedo documentation Use them to dynamically reconfigure the domain dynamically resize the domain start/shutdown extra servers monitoring scripts tmadmin -r for read only mode

Java Administrative Applet

Conclusion So now you know BEA Documentation BEA Tuxedo/Jolt is a messaging protocol How the messages are moved around How PeopleSoft has used it How the PIA has been constructed Options for Tuning and Sizing Load Balancing Multi-Node domains Monitoring Scripts BEA Documentation is on your CD http://e-docs.bea.com

More Information about PT8 Application Server PeopleSoft 2000 Conference: Americas 8006: PeopleTools 8.1 Application Server 8010: Inside the Application Server Buzz Voelker PeopleTools Application Server Team

Final Caveat PeopleSoft will not support any of the changes that I have described in this presentation Everything described in this presentation has been done, in production, at UBS If you want to use a multi-node domain you are strongly recommended to arrange support/consultancy direct from BEA.

Questions?

PeopleTools Application Server & BEA Tuxedo Internals david.kurtz@go-faster.co.uk Go-Faster Consultancy Ltd. http://www.go-faster.co.uk