1. Optimal Performance When Running CICS in a Shared LPAR Environment
IBM Advanced Technical Support
Optimal Performance When Running CICS in a Shared LPAR Environment
Kathy Walsh
IBM
Washington Systems Center
Project: CICS Session: xxxx
© IBM Corporation 2003
Foil #

2. Disclaimer and Trademarks
The information in this document has not been submitted to any formal IBM test and is distributed on an "as is" basis without any warranty expressed or implied. Use of this information or the implementation of any of these techniques is a user responsibility and depends on the user's ability to evaluate and integrate them into the user's operational environment. While each item may have been reviewed for accuracy in a specific situation there is no guarantee the same or similar results may be achieved elsewhere. Users attempting to adapt these techniques to their environments do so at their own risk.
The following names are trademarks of the IBM Corporation and may be used throughout this presentation:
zSeries, CICS, CICS/ESA, CICS/MVS, COBOL/2, COBOL/370 DB2, DFSMS, DFSMS/MVS, z/VM, z/OS, IBM Logo, Enterprise System/9000, e-business Logo, Enterprise Systems Architecture/390, ES/3090, ES/9000, ESA/370 Enterprise Systems Connection Architecture, ESA/390, ESCON Hiperbatch, Hipersorting, Hiperspace, IBM, IMS, IMS/ESA, MVS, MVS/XA, Language Environment, MVS/DFP, MVS/ESA, MVS/SP, NetView, PR/SM, RACF, REXX, Processor Resource/System Manger, RAMAC, RMF, S/370, S/390, 3090, Large Systems Performance Reference, LSPR, OS/390

3. Agenda
This session will review basic performance principles when running CICS in a shared LPAR environment. Performance impacts of running CICS on fewer, faster CPs will be discussed in detail. The session will review the LPAR definitions and their impact on performance and throughput. Issues involving dispatching priorities as set by WLM and known performance concerns when doing either server or region consolidation will be reviewed.

4. zSeries Virtualization via PR/SM Technology
LPAR 2
LPAR 1
LPAR 3
LPAR n
Up to 30 LPARs
PR/SM
z/OS
OS/390
zLINUX
1 to 30 LPARs per CEC
Operating Systems don’t know they are not running directly on the hardware
PR/SM ™ is managing the resource allocations based on installation controls

5. Partitioning Controls
The number of partitions and their relative weights
The number of logical CPs defined to the partitions
The ratio of logical CPs to physical CPs defined
The effect of the partitions shared weight and any impact of capping on the partition
The CP usage; either general purpose, traditional CPs or the use of IFL/ICF CPs
The partition mode, either dedicated or shared
The type of system control program (z/OS, z/VM, Linux, etc.), and the workload characteristics in each partition

6. Important Terms to Understand
LPAR weight and per CP share
Effective Dispatch Time
Partition Dispatch Time
Short CPs
Important Concepts to Understand
LPAR weights become important only when the processor is very busy or capped
There are two dispatchers involved in making resource allocations
PR/SM
Operating System

7. RMF Partition Report 2084-309 = 325 MIPS/CP
P A R T I T I O N D A T A R E P O R T
MVS PARTITION NAME WSCWRC1
NUMBER OF CONFIGURED PARTITIONS
NUMBER OF PHYSICAL PROCESSORS CP
ICF
WAIT COMPLETION NO
DISPATCH INTERVAL DYNAMIC
PARTITION DATA PROCESSORS
NAME STATUS WEIGHTS CAPPING NUM TYPE
WSCWRC A NO CP
WSCWRC A NO CP
This is a copy of the RMF partition data report. It will show the number of physical CPs installed on the processor as well as the number of logical CPs and LPAR weight for each partition.
This report can be used when we are trying to understand the potential for short CPs.
This report shows the processor has 9 physical CPs with no ICFs. There are two partitions, each has 9 logical CPs defined. The weight of WSCWRC1 is 800 and WSCWRC2 is 200.
These values will be used on up coming foils
Later examples will use a Z97 with a value of 151 MIPS/CP
= 325 MIPS/CP

8. Effective Dispatch Time
Measurement which states the busy of the logical CPs
Independent measure of capacity
Can run out of logical CP capacity before the processor is 100% busy
Partition Dispatch Time
Measurement of the partition busy in processor terms
Differs from effective time when the number of logicals defined to the partition does not match the number of general purpose CPs
It is this metric which is used in capacity planning exercises
PARTITION DATA
----MSU CAPPING-- PROCESSOR-
NAME S WGT DEF ACT DEF WLM% NUM TYPE
OSP A NO CP
OSP A NO CP
OSP A NO CP
OSP A NO CP
CF A DED CP
CF A DED CP
*PHYSICAL*
TOTAL
-- AVERAGE PROCESSOR UTILIZATION PERCENTAGES --
LOGICAL PROCESSORS --- PHYSICAL PROCESSORS ---
EFFECTIVE TOTAL LPAR MGMT EFFECTIVE TOTAL

9. Calculate LPARs Weight (relative value)
LPAR Weight
Share =
Sum of Weights
WSCWRC1 share = 800/1000 = 80%
WSCWRC2 share = 200/1000 = 20%
This foil shows how the weight is used by LPAR to determine how much of the CPU is guaranteed to each partition. These values will be used to determine the number of MIPS available to the logical CP
TIP
All active LPARs are used even if an SCP is not IPL'ed
Only Shared LPARs are used

10. Calculate amount of processor guaranteed to each LPAR
# of General Purpose (GP) Physical CPs (PCP) * LPAR share
WSCWRC1 capacity = 9 * .80 = 7.2 CPs
WSCWRC2 capacity = 9 * .20 = 1.8 CPs
TIP
The processor guarantee is used to offer protection to one LPAR over other busy LPARs demaning service
Once we know the amount of processor capacity guaranteed to the LPAR, we need to determine the amount of capacity available on a CP basis as LPAR manages the processor at the physical CP level.
Since the machine has 9 physical General Purpose (GP) CPs and WSCWRC1 is guaranteed 80% of the processor, it is guaranteed 7.2 CPs of capacity.

11. Determine Per CP Fair Share Dispatch Time
Guaranteed Processor Value
Partition dispatch time % = * 100
# LCPs in the partition
WSCWRC1 = 7.2 / 9 = 80% or .8 * 325 = 260 MIPS
WSCWRC2 = 1.8 / 9 = 20% or .2 * 325 = 65 MIPS
Alternative Is:
WSCWRC1 = 7.2 / 8 = 90% or .9 * 325 = 293 MIPS
WSCWRC2 = 1.8 / 2 = 90% or .9 * 325 = 293 MIPS
The number of MIPS available to a logical CP is determined by the number of physical CPs guaranteed to the partition divided by the number of logical CPs defined to the partition.
Using the WSCWRC1 partition as an example, since it is guaranteed 7.2 CPs of capacity and it has 9 logical CPs defined, the logical CP will have access to the physical CP 80% of the time.
If the physical CP can deliver 151 MIPS, and the logical CP can access the physical CP 80% of the time 151 x 80% = 121 MIPS. This means any work running on WSCWRC1 will see a CP which is delivering 121MIPS.
Using the same calculations, WSCWRC2 will deliver 30 MIPS per logical CP
TIP
Biggest per CP Share possible is best when processor is busy
TIP

12. What Are 'Short CP's? Term created by the WSC performance staff
Performance phenomenon created by LPAR hipervisor enforcing LPAR weights on busy processors or capped partitions
LPAR ensures each partition has access to the amount of processor specified by the LPAR weights
This can reduce the MIPS delivered by the logical CPs in the partition
Controlled by a combination of LPAR weights and number of Logical CPs
Potential Performance Problems
In a processor migration “short CPs” are not a problem as long as the partition on the new CEC has access to an equal or greater number of MIPS per CP
Techdocs Item: WP – Performance Considerations when moving to Fewer, Faster CPUs
The issue of 'short CPs' is there can be a difference between the number of MIPS available on the physical CP and the number of MIPS available to the logical CP(LCP). It is possible for the logical CP to have access to less MIPS then the physical CP (PCP) can deliver. This phenomenon is a function of the number of logical CPs defined to the partition as well as the weight guaranteed to the partition. The problem arises when the weight is a low percentage while the number of logical CPs is high.

13. WSC 'Short CPs' Possible Performance Symptoms
Prod CICS can't keep up and transactions are backing up
Production system is 'sluggish'
High performance address space may not be getting enough cycles (GRS, XCF, etc.)
Test system is not processing XCF requests in a timely fashion and production system is experiencing the performance problem in a Sysplex (sympathy sickness)
GRS on production system
Catalog processing
Due to logical CP losing access to physical CP
MVS is NOT AWARE the CP is gone
High priority task doesn't have a physical assigned, while the low priority task does have a physical CP assigned
These are the symptoms people report which often tie back to 'short CPs'.
A high priority address space like CICS seems to be starved for cycles, when lower priority work is running.
The system address spaces all have high dispatch priorities and expect to get access to the processor when needed. If they can't get access to the physical CP, they start slowing down which makes the entire MVS image sluggish.
Sysplex causes multiple systems to communicate with each other. If the test system is part of the production Sysplex and the test system is suffering 'short CPs', it gets sluggish which can cause problems for the system which must communicate with it, including the production systems.
LPAR honors the weight requests specified by the user on a busy system. 'Short CPs' happen because the MVS dispatcher does not know it's logical CP has lost access to the physical CP. This reduces the effectiveness of the MVS dispatcher.

14. MVS Dispatcher when LPAR Weights are being Enforced
Interval
CP 0
CP 1
CP 2
CP 3 1
CICS,STC,Batch,Batch
CICS
L=P
BATCH
STC 2
CICS,STC,Batch L 3
CICS,Batch,Batch,Batch 4
CICS Active :4 = 100%
CICS Dispatched :4 = 50%
LPAR BUSY 10:16 = 63%
MVS BUSY 12:16 = 75%

15. Environment Both CECs are the same technology
IMS Data Sharing workload runs on 2 images on different CECs
Both CECs are the same technology
Both LPARs run the same, exact transactions
IMS response time on SYB is consistently higher than SYA
Work is defined with high importance, and a stringent goal
09.00
09.10
09.20
09.30
09.40
09.50
10.00
10.10
10.20
10.30
10.40
10.50
11.00
Hour
0.5 1
1.5 2
2.5 Ti
SYA - RT
SYB - RT
Response Time

16. Short CPs - An Example SYB has short CPs (MVS Busy / LPAR BUSY)
Each logical CP was allowed 18% of a CP across 6 CPs
Change number of logicals to 2, get the CP share to 54%

17. Short CPs
Short CP Ratio dropped to approx 1.5 and response times dropped noticeably

18. Impacts on Throughput 4 partitions whose total weight = 410
Partition A has a weight of 110, or relative weight of 27%
Partition A has 4 logical CPs defined
Per CP share is: (.27 * 10 CPs = 2.7 CPs / 4 CPs = 68% per CP
Change Partition A to have 3 logical CPs defined
Per CP share is: (.27 * 10 CPs = 2.7 CPs / 3 CPs = 90% per CP

19. Short CPs Possible Impact to throughput and transaction response times
Approaches to curing "short" CPs
Reduce the number of logical CPs
Increase the partition's weight
Implement IRD
WSC Hint and Tip on Techdocs
TD101238: zSeries Performance: Determining the Logical CP Requirements for a Partition

20. Causes of LPAR Overhead
Now lets talk about the issues of LPAR overhead as a function of the LPAR definitions
© IBM Corporation 2003
Foil #

21. Logical to Physical Ratio
Management of the Hardware
LPAR must assign a physical CP to a logical CP in order to execute instructions
CPU cost in the *PHYSICAL partition on RMF Partition Report
zSeries Processor
Partition 1
Partition 2
Partition 3
LCP
LCP
LCP
LCP
LCP
Most customers run their processors in LPAR mode and share the logical CPs across all the partitions. LPAR is responsible to assign a physical CP to a logical CP when the logical CP has work to do.
It takes cycles for LPAR to do this management function.
An estimate of this cost is reported on the RMF Partition Report on the line with a partition name of *PHYSICAL* This is not the total cost of LPAR.
IBM has a tool which can be used by the IBM account team to estimate the cost of LPAR management. This cost is most effected by the number of logical CPs defined and the number of physical CPs in the shared pool
This table shows the projected cost of different LPAR configurations as reported by the LPAR/CE tool.
As can be seen, as the logical to physical ratio increases, so does the LPAR management overhead
PCP
PCP
Logical to Physical Ratio
LPAR Overhead
1:1
0.22%
2:1
0.42%
3:1
0.55%
10:1
0.81%

22. Management of the Partition
Basic Mode
z/OS
Hardware
PGM
STIDP
z/OS
LPAR Mode
Hardware
LPAR
PGM
STIDP
LPAR must get involved with certain instructions
Cost reflected in LPAR MGMT Column on RMF Partition Report for each partition
There is another cost of LPAR, the cycles needed to support the operation of a partition.
There are instructions which communicate with the hardware when a machine is running in BASIC mode.
The LPAR hipervisor is involved in processing these instructions when a system is running in LPAR mode. These instructions can often be machine state type instructions and each LPAR may have it's own state.
The CPU time consumed by LPAR to help process these instructions is reported in the LPAR Management column of the Partition Data Report
PARTITION DATA --- AVERAGE PROCESSOR UTILIZATION PERCENTAGES --
LOGICAL PROCESSORS --- PHYSICAL PROCESSORS ---
NAME S WGT DEF EFFECTIVE TOTAL LPAR MGMT EFFECTIVE TOTAL
SYS6LP01 A
SYS6LP02 A
*PHYSICAL*
TOTAL

23. High Speed Buffer Contention
The high speed buffer is 'fast' memory
Accessing data from the high speed buffer improves the speed of the PCP
Data not found in the high speed buffer reduces the effective speed of the PCP
Each time a new LCP is associated with a PCP, increased risk of HSB miss
The impact of the HSB miss is not reported in RMF, but is reflected in increased TCB time for jobs
LPAR/CE includes estimated TCB time elongation
PCP
HSB
LPAR
Memory
LCP
LCP
The speed of a processor is affected by the effectiveness of the High Speed Buffer. The higher the hit ratio of the HSB, the faster the machine. The lower the ratio, the slower the machine.
A major contributor to the hit ratio of the HSB is the contention for the HSB. The more units of work running, the more the potential contention of the HSB.
Since the HSB is associated with the physical CP, adding more logical CPs increases the contention of the physical CP HSB. This has the potential to slow down the processor.
This effect is not reported as a separate field in RMF. Instead, it is surfaced as increase TCB time for running jobs.
This value is not reported in RMF, but is estimated by the IBM LPAR/CE tool.
LCP

24. Other LPAR Performance Factors
Now lets talk about the issues of LPAR overhead as a function of the LPAR definitions
© IBM Corporation 2003
Foil #

25. Logical to Physical CP Ratio
Strive to keep logical to physical ratio in the 2:1 or 3:1 area
A higher ratio will work but will cause an increased cost which needs to be factored into the capacity plan
Biggest issue to reducing the logical to physical CP ratio is the requirement to run small LPARs as z/OS uni-processors
Availability issues of running z/OS as a uni-processor
Places greater emphasis on doing LPAR consolidation to make fewer LPARs which need more than 1 CP of capacity
Virtual storage constraints need to be reviewed

26. Type of CP Usage ICF Impact IFL Impact
Within 10% of the performance of a stand-alone CF of the same processor family
Impact on SCP Engines
Addition of an ICF CP showed little to no degradation to the z/OS CPs
Impact of adding an ICF would be <= 1% per ICF
Example: Using a , rated at 2932 MIPS, the addition of an ICF would cause the new configuration to be rated at 2903 MIPS. This is significantly closer to the non-ICF rating than having a 10-way configuration which is rated at 2875 MIPS
IFL Impact
Depends upon the application activity level, impact can be slight (<10%) up to the full n-way impact of an additional CP

27. IRD Overview Four components
CPU weight management
Vary CPU management
Dynamic CHPID management
Channel subsystem I/O priority management
Allows system to dynamically move resources to the work
Introduces the concept of the LPAR cluster

28. LPAR Cluster 2064 2064 LPAR Cluster
All z/OS images in the same sysplex on the same z/Series processor
2064
2064
z/OS
SYSPLEX1
LPAR
Cluster
z/OS
SYSPLEX1
z/OS
SYSPLEX2 CF
z/OS
SYSPLEX1

29. CPU Weight Management
WLM manages physical CPU resources across z/OS images within an LPAR cluster based on service class goals
Dynamic changes to the LPAR weight
Sum of LPAR weights can be redistributed within the LPAR cluster
Partition(s) outside the cluster are not affected
Move CP resource to the partition which needs it
2084
z/OS
SYSPLEX1
LPAR
Cluster

30. Vary CPU Management
Dynamic management of online CPs to each partition in the LPAR cluster
Optimizes the number of CPs for the partition's current weight
Prevents 'short' engines
Maximizes the effectiveness of the MVS dispatcher
2084
z/OS
SYSPLEX1
z/OS
SYSPLEX1
Dynamic
Config On/Off
Dynamic
Config On/Off
LCP
LCP
LCP
LCP
LCP
LCP
PCP
PCP
PCP
PCP

31. Summary
Partition controls need to be well understood to ensure optimal performance
Impact of short CPs on high importance workloads can be very noticeable
Can be successful running a large number of LPARs on a single CEC though the capacity impacts need to be evaluated
True even when moving from more, slower CPs to fewer, faster CPs
ROT: Ensure LPARs on the new CEC has access to as many or more MIPS than on the previous CEC
Evaluate the use of IRD to provide autonomic capability

32. Time for...