1. PeopleTools Application Server & BEA Tuxedo Internals
Go-Faster Consultancy Ltd.

2. Who am I? Independent Consultant Performance Tuning
PeopleSoft / Oracle

3. Rules of Engagement Interrupt me:
if you think I have got something wrong
if you have a question
if you can’t hear me
I put this slide in to make sure I say this

4. BEA Tuxedo Historical Overview Tuxedo Internal Architecture
PeopleSoft 3-tier client implementation
How does PeopleSoft Internet Architecture (PIA) fit in
Configuration issues
Monitoring

5. Brief Historical Background
In the beginning
monolithic
Presentation
Business Logic
Data Access (DBMS)
IBM Mainframe

6. Client Server Client Server LAN Two-tier client-server SQL
Presentation
Business Logic
Data Access (DBMS)
SQL
Raw Data

7. 3-Tier Client Server Client Server LAN “1st Generation” GUI +
Application
Logic
Data Access (DBMS)
Business Logic
Client
Server
LAN
Request
Response

8. 3-Tier Client Server “2nd Generation” - Tuxedo Business Logic
Data Management
TUXEDO Servers
Presentation
Logic
TUXEDO Clients
DBMS 1
...
DBMS n
Database Servers
Service 1
Service 2
Service n
TUXEDO
Messaging
SQL

9. Where did Tuxedo come from?
Bell Labs
Transactions for
UniX
Extended for
Distributed
Operations
The key words here are distributed operations. Different parts of a transaction occur on different nodes

10. What is Tuxedo? It is the knots on the ends of the string!
Imagine a telephone make of string and tin cans. Every child has made one
If a tuxedo application where that telephone, the tuxedo bits would be the knots on the end of the string

11. PeopleSoft 2-Tier Client
Presentation
Logic
Panel
Load
Save
Field
Change
etc...
PeopleSoft windows client is written in C
Imagine cutting up the two tier client

12. PeopleSoft 3-Tier Client
Presentation
Logic
Tuxedo Client
Tuxedo Server
Panel
Load
Save
Field
Change
etc...
Client Side
Server Side

13. PeopleSoft 2-Tier Client
Client Connects Directly to the Database
1 database connection per window
inactivity timeout
DATABASE

14. PeopleSoft 2-Tier Client
Client Connects Directly to the Database
1 database connection per window
inactivity timeout
What happens in 3-Tier?
DATABASE

15. PS/Tuxedo Internal Structure
1st Application Server Process Started
Bulletin Board Liaison (BBL)
Reads PSTUXCFG configuration file
Shared Memory Segment
Bulletin Board (BB) BB
DATABASE
PSTUXCFG
BBL

16. PS/Tuxedo Internal Structure
Workstation Listener (WSL)
Listens for connection request from clients on a specified port
Client initially contacts the WSL
WSL BB
DATABASE
BBL

17. PS/Tuxedo Internal Structure
Workstation Listener (WSL)
Client needs to know where to find the WSL
Machine and port specified in Configuration Manager
WSL BB
BBL

18. PS/Tuxedo Internal Structure
Workstation Handler (WSH)
handles traffic from client after connection
Client is assigned to a port on the WSH
WSH port range can be constrained to match a firewall
Additional WSH can be spawned if necessary
Compress Messages
WSL
WSH BB
DATABASE
BBL

19. PS/Tuxedo Internal Structure
PSAUTH
AUTHQ
PSAPPSRV
APPQ
PSQCKSRV
QCKQ
PSQRYSRV
QRYQ
PSAPISRV
APIQ
PSSAMSRV
SAMQ
WSL
WSH BB
DATABASE
BBL

20. How does it work? BB PSAUTH AUTHQ PSAPPSRV APPQ PSQCKSRV QCKQ PSQRYSRV
QRYQ
PSAPISRV
APIQ
WSHQ
PSSAMSRV
SAMQ
WSL
WSH BB
DATABASE
BBL

21. How does PIA work? Java Client & PIA use Jolt Java Class Library
Jolt Listener maps Java functions to Tuxedo Messages
BEA Jolt connects Java clients to applications built using the BEA TUXEDO system. The TUXEDO system provides a set of modular services, each offering specific functionality related to the application as a whole.
For example, a simple banking application might have services such as INQUIRY, WITHDRAW, TRANSFER, and DEPOSIT. In PeopleSoft it is the functionality of the client
Typically, service requests are implemented in C or COBOL as a sequence of calls to a program library.
The Jolt Server implementation acts as a proxy for the Jolt client, invoking the TUXEDO service on behalf of the client. The BEA Jolt 1.2 Server accepts requests from the Jolt clients and maps those requests into TUXEDO service requests.
Diagram is taken from BEA Jolt Manual

22. Jolt Internal Structure
PSAUTH
AUTHQ
PSAPPSRV
APPQ
PSQCKSRV
QCKQ
PSQRYSRV
QRYQ
JSHQ
PSAPISRV
APIQ
JSL
JSH
JREPSRV
PSSAMSRV
SAMQ
WSL
WSH BB
JREPOSITORY
DATABASE
BBL

23. New structure for PIA (Apache Webserver)
Sessionid = “xy1k…vm8010
http 1.1
address+cookie
(stateless)
Cookie
Java
JVM
Servlet Engine
Servlet thread
IS stateful
Servlet Container
Servlet
Webserver
Servlet
Thread
Servlet
Thread
HTTPD
Client
Servlet
Thread
Client
Disk
(static
objects)
mod_jserv
Port 8010
Tuxedo
Application
Server
JSL
JSH
JSH
JSH

24. What is the difference between the web servers?
Apache
Separate Servlet Engine
Separate encryption module (Raven)
Weblogic
Contains an integral servlet container

25. Progressive Connection Concentration
35000 users
Typical picture
35,000 users
1000 concurrent connections
1000 users active in the last n minutes
where n is the Java + JSH timeout
therefore need 1000 threads
but maybe only 4 JVMs
100 JSH
assumes 10 clients per JSH
10-20 PSAPPSRV processes
each PSAPPSRV connects to database
1 database
1000 connections
4 JVMs
1000 Java Threads
100 JSHs
10-20 PSAPPSRVs
1 database

26. Configuration psappsrv.val psadmin (configuration dialogue)
psappsrv.cfg
ubbgen
(PS Utility)
psappsrv.ubx
psappsrv.ubb
tmloadcf
(tuxedo
process)
PSTUXCFG
BBL
(domain
startup)

27. Configuration Files psappsrv.cfg psappsrv.val psappsrv.ubx
contains configurable parameters
psappsrv.val
validation file for configurable values
psappsrv.ubx
configuration template for Tuxedo domain
psappsrv.ubb
configuration file for Tuxedo domain
PSTUXCFG
compiled configuration file
read by BBL at domain startup

28. psappsrv.ubx -v- psappsrv.ubb
psappsrv.ubx & psappsrv.cfg
PeopleSoft file
make changes to these files
difficult to read
psappsrv.ubb
Tuxedo file
never make changes to this file
easier to read
this file is compiled by tmloadcf to generate domain configuration file PSTUXCFG
variables fully resolved

29. psappsrv.ubb Five (or Six) Sections Resources Machine Groups Servers
resource limits -> BB sizing
Machine
Resource limits. Environment
(Network
Connection between Machines)
Groups
Logical Split (BASE, APPSRV, JREPGRP, JSLGRP)
Servers
Number of servers, Command line
Services
Timeout, Load, Priority

30. Application Server Configuration Issue
Move quick PSAPPSRV services into a second server (PSQCKSRV) (y/n)?
<=PT7.58 this question is misleading
services are additionally advertised on PSQCKSRV as well as PSAPPSRV
so they are advertised on more than one queue
>=PT7.59 and PT8.1
services are de-advertised from PSAPPSRV
quick services not used in PIA, only IC* services
so not a problem

31. PeopleTools <=7.58 Quick Service BB PSAPPSRV APPQ PSQCKSRV QCKQ WSL
WSH BB
DATABASE
BBL

32. Inside the Bulletin Board
Without Load Balancing (default)
Table of servers process where each service is advertised
Read and populated as a stack
WSL looks for first free server where request can be enqueued
This can cause problems
PSAPPSRV
PSQCKSRV
PSQCKSRV
PSQCKSRV
PSAPPSRV
PSAPPSRV
PSAPPSRV

33. Load Balancing
Load Balancing is about choosing the queue upon which to place the message
by default all requests have the same load 50
PSAPPSRV
APPQ 50
PSQCKSRV
QCKQ

34. Load Balancing Trace Add -r to application server command line
CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S PSAPPSRV”
-r enables trace to file ‘stderr’ (not channel 2)
-e overrides the output filename
process trace file with Tuxedo utility ‘txrpt’
obtain average service time
make load proportional to average service time

35. Load Balancing Trace Output from TXRPT SVCNAME 18p-19p TOTALS
Num/Avg Num/Avg
SqlRequest / /0.10
MgrClear / /0.04
RamList / /0.43
SamGetParmsSvc 1/ /0.50
TOTALS / /0.12

36. Load Balancing load = execution time (seconds) x 100 78 386 135 599 50
PSAPPSRV
APPQ
599 50 63 3 7 6 30
PSQCKSRV
QCKQ
116

37. Multiple Queues on Busy System
You can have many server processes on a single queue
BEA recommend no more that 10 server processes per queue
If you have queue contention it is likely to impede the WSH/JSH processes enqueuing requests. You are unlikely to observe queuing with PQ.
So, if you want more than 10 PSAPPSRV processes you should use two APPQ queues
In, which case you should to enable load balancing and you must calculate the load.
Less significant with PIA because services different.

38. Multiple Queues on Busy System
PSAPPSRV
APPQ1
PSAPPSRV
APPQ2
WSL
WSH BB
DATABASE
BBL

39. Multiple Queues on Busy System
psadmin will not correctly calculate
MAXACCESSERS
MAXSERVICES
tmadmin->bbs
will report maximum number of services
count of all services advertised on each server process
and if you have multiple work station listeners
MAXWSCLIENTS

40. Service Priority Priority is about queue jumping
servers de-queue the request
9 out of 10 de-queued on the basis of priority
1 out 10 is taken from the front of the queue 80 50
PSQCKSRV
QCKQ

41. Service Priority All PeopleSoft Services have same priority
specified in SERVICES section in psappsrv.ubb
therefore there is no queue jumping
no reason to believe that it would be advantageous

42. Tuning Tip Financials On-line journal edit & post - remote call
Service initiates a Cobol process
Process can take minutes
RemoteCall services can block APPQ
Move RemoteCall service to a separate queue

43. Multi-Node Domain 1 domain on many nodes BB BB DBBL WSL WSH BBL
PSAPPSRV
APPQ
BRIDGE
TLISTEN
BRIDGE
WSL
WSH
BBL BB
PSAPPSRV
APPQ
DATABASE
TLISTEN

44. Multi-Node Domain Distinct Bulletin Board Liaison process
Master Node
TLISTEN process
Listens for incoming requests
BRIDGE
Sends & Receives messages between nodes
in a load balancing environment
configurable bridge load add to remote queues
limiting number of messages sent across the bridge

45. Multi-Node Domain DO ATTEMPT THIS WITHOUT BEA CONSULTANCY At UBS
PT 7.5x
susceptible to network errors
integrated into HP service guard
ultimately this was removed from production because of transient network errors between the nodes causing the domain to ‘partition’

46. So what is going on? tmadmin Tuxedo command line interface
also available within psadmin

47. tmadmin commands pq - printqueues amount of work queued and waiting
not always accurate

48. tmadmin commands psr - printservers
reports activity of individual server processes

49. tmadmin commands
psc - printservices
each service on each server

50. tmadmin commands Lots of other tmadmin commands
documented in the Tuxedo documentation
Use them to
dynamically reconfigure the domain
dynamically resize the domain
start/shutdown extra servers
monitoring scripts
tmadmin -r for read only mode

51. Java Administrative Applet

52. Conclusion So now you know BEA Documentation
BEA Tuxedo/Jolt is a messaging protocol
How the messages are moved around
How PeopleSoft has used it
How the PIA has been constructed
Options for Tuning and Sizing
Load Balancing
Multi-Node domains
Monitoring Scripts
BEA Documentation
is on your CD

53. More Information about PT8 Application Server
PeopleSoft 2000 Conference: Americas
8006: PeopleTools 8.1 Application Server
8010: Inside the Application Server
Buzz Voelker
PeopleTools Application Server Team

54. Final Caveat
PeopleSoft will not support any of the changes that I have described in this presentation
Everything described in this presentation has been done, in production, at UBS
If you want to use a multi-node domain you are strongly recommended to arrange support/consultancy direct from BEA.

55. Questions?

56. PeopleTools Application Server & BEA Tuxedo Internals
Go-Faster Consultancy Ltd.