Download presentation
Presentation is loading. Please wait.
Published byMaria Vitória Bento Fidalgo Modified over 6 years ago
1
Understanding the AIX Performance Data in a PowerVM Partition
Pete Weilnau Philadelphia CMG Feb 29, 2008 © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
2
Understanding the Instrumentation Putting it into practice
Agenda Review of PowerVM Instrumentation Understanding the Instrumentation Putting it into practice © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
3
Disclaimer All trademarks referenced in this presentation are the properties of their respective companies. Some diagrams are from IBM Redbook SG Advanced POWER Virtualization on IBM System p5: Introduction and Configuration © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
4
Available on System p5 and higher
PowerVM Available on System p5 and higher A marketing umbrella term for a number of hardware and software technologies: Dynamic LPAR (DLPAR) Shared Processor Pool (Micro-partitioning) Virtual IO Server (VIO) Simultaneous Multi-Threading (SMT) Virtual LAN (VLAN) Integrated Virtualization Manager (IVM) Capacity on Demand (CoD) Live Partition Mobility (POWER6 only) Shared Dedicated Capacity (POWER6 only) we are going to focus mainly on APV tech It includes support for LPARs that run AIX, Linux (RH or SuSE), or even i5 optional on the smaller systems, standard on the bigger systems IBM Virtualization Engine is an (older?) umbrella marketing term for these features? WPARs - workload partitions © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
5
System p overview intranet remote admin AIX LPAR Linux LPAR VIO Server
public access to HMC private network between HMC and managed system(s) managed system LPARs may also have public access the whole System p server is a managed system or CEC (Central Electronic Complex). “Managed system” in that it is managed from the HMC more than one system can be managed from the HMC PerfMan has used the term “managed system” for many years to mean something else - an operating system image that is reporting metrics to the PerfMan system. For UNIX/Linux, this means it is running the PerfMan for UNIX Agent. We will be careful to distinguish which meaning we intend during this discussion HMC Hypervisor firmware © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
6
always enabled (think “NO BASIC MODE”) Key features
POWER Hypervisor System firmware introduced w/ POWER5 always enabled (think “NO BASIC MODE”) Key features Dynamic LPAR (DLPAR) allows dynamic reallocation of resources ... but only if the OS supports it! shared processor pool (Micro-partitioning) AIX 5.3 & Linux only (no i5/OS support) allows fractional physical CPU entitlement virtual device support (VLAN, SCSI, consoles) provides security isolation between LPARs “micro” is a misnomer, the granularity is by multiples of 1/10th for LPAR initial entitlements multiples of 1/100th for adjustments at runtime may differ by hardware © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
7
POWER Hypervisor source: SG24-7940-02
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) source: Advanced POWER Virtualization on IBM System p5 – SG
8
Virtual IO Server a software appliance that runs in a dedicated LPAR
as the name implies, it provides virtual IO services to other (“client”) LPARs virtual disks virtual Ethernet adapters shared Ethernet adapters (SEA) Integrated Virtualization Manager (IVM) feature added w/ VIO Server 1.2 provides a browser-based subset of HMC function main restriction – all resources are owned by VIO, no dedicated resources you may be able to avoid buying an HMC if you only have a few CECs or lightweight needs document restrictions of IVM? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
9
Hardware Management Console (HMC)
hardware appliance to manage one or more managed systems (CECs) includes partition management and many other management functions provides optional web interface for remote access CEC – Central Electronic Complex © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
10
Web-based System Manager (WebSM)
WebSM provides a Java-based subset of HMC function © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
11
Processor allocations
properties of the “managed system” – the CEC (Central Electronic Complex) or hosting server physical hardware platform © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
12
SMT and PURR SMT = Simultaneous Multi-threading multitasking – software & hardware invention that allows multiple operating system tasks to appear to run simultaneously, maximizing use of the CPU by interleaving IO and CPU to avoid IO delays multithreading – hardware invention that allows multiple operating system threads to appear to run simultaneously, maximizing use of the core execution unit by interleaving internal CPU operations to avoid internal delays (e.g. cache & memory access) PURR = Processor Utilization Resource Register These are hardware counter(s), two per physical processor, one per logical processor (thread) the hypervisor virtualizes the PURR for each virtual processor dispatched to the shared processor pool Designed to give better CPU dispatch cycle counting & utilization metrics in an SMT and/or shared processor environment by state/event, not sampling This is an improvement over the pre hz timer-based sampling method. The whole 10ms tick was assigned to whoever was dispatched when the timer popped. This was never that accurate, but it's really lousy when SMT is enabled or when shared processors are in use. All AIX 5.3 accounting, performance commands, and WLM code exploit the PURR Linux still uses the tick, but also reads the PURR and makes it available in /proc AIX commands support access to correctly virtualized metrics and some hypervisor metrics sar, vmstat, iostat, mpstat, ps lparstat topas curt © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) source: Advanced POWER Virtualization on IBM System p5 – SG
13
Physical, virtual, and logical processors
source: Advanced POWER Virtualization on IBM System p5 – sg Remember that APV only supports LPARs with either dedicated or shared processors – can’t have some of each at the same time. Physical processors are allocated by the server’s firmware and hypervisor, under direction from the HMC. They can either be dedicated to an LPAR or belong to the shared processor pool. When processors are dedicated, the LPAR’s operating system sees 1 logical CPU per physical CPU. (not shown – if SMT is on in firmware and OS, each physical processor may have 2 threads and is thus seen as two logical CPUs) When processors are shared, the LPAR (micropartition) may be defined with a fractional share of a CPU. It may also be capped, or uncapped with an associated weighting. Virtual processors are a part of the LPAR definition for a micropartition. They represent a firmware abstraction layer between the operating system and the shared processor pool. They are allocated in whole units, not fractional units, and they hide the details of fractional, constantly changing processor allocations from the operating system scheduler/dispatcher. The firmware evenly spreads the LPAR’s CPU entitlement among the virtual processors. The minimum allocation per virtual processor is model dependent (our is 0.10) and the maximum is 1. Therefore, if your entitlement is 3.5 processors but you’ve only got 3 virtual processors, you can never get more than 3 processors worth of work done. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) source: Advanced POWER Virtualization on IBM System p5 – SG
14
Focus of this research:
What can we learn about LPAR level activity from an AIX Point of View? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
15
Let’s talk Instrumentation
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
16
iostat, vmstat, sar and time:
Old Faithfuls iostat, vmstat, sar and time: Use PURR based statistics whenever SMT is enabled %user, %system, %iowait and %idle. When executing on a shared-processor partition: pc or %physc represent % physical processor consumed. ec or %entc represent % of entitled capacity consumed. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
17
% of what? Isn’t it a moving target?
iostat example What does %user, %sys, %idle, %iowait really represent in a shared-processor environment? % of what? Isn’t it a moving target? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
18
When executing on a shared-processor partition:
Old Faithfuls Sar –p ALL: Use PURR based statistics whenever SMT is enabled %usr, %sys, %wio and %idle. When executing on a shared-processor partition: physc represent % physical processor consumed. %entc represent % of entitled capacity consumed. lcpu = shows number of logical CPUs ent = shows entitlement © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
19
What does that ‘U’ record represent?
Sar –P ALL example What does that ‘U’ record represent? ‘-’ normally shows the total © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
20
sar –P ALL U row is reported if running in shared processor pool.
It shows the UNUSED system-wide capacity. System-wide = LPAR. If in uncapped mode: The individual processor rows (phyc) are calculated against physical consumption Unused is computed against entitlement and not physical consumption. Source: man sar © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
21
Displays both virtual and logical processors
Old Faithfuls mpstat -s: Displays both virtual and logical processors proc0 – represents a Virtual CPU cpu0 and cpu1 represent logical CPUs © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
22
cpu0 and cpu1 represent logical CPUs
Other Commands Mpstat –a Provides a lot of detail info about processor activity, well beyond the scope of this presentation. cpu0 and cpu1 represent logical CPUs © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
23
lparstat :monitoring mode lparstat –h :hypervisor summary
lparstat - New lparstat: Shows configuration and performance information for the partition in which it is run. 4 Modes: lparstat :monitoring mode lparstat –h :hypervisor summary lparstat –H :hypervisor calls lparstat –i :Configuration info Lparstat command appears in AIX 5.3 even when on backlevel PowerPC hardware. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
24
lparstat Psize = processors in shared pool
Lbusy = percent occupation of the logical CPUs at the system and user level Vcsw = virtual context switches Phint = phantom interrupts that the partition received A phantom interrupt is an interrupt targeted to another partition that shares the same physical processor. For example, one partition starts an I/O operation. While the partition is waiting for the I/O to complete, it cedes the physical processor to another partition. The I/O operation completes and the controller sends an interrupt to the requesting processor, but as the interrupted partition running is not the intended destination, the partition says “not for me” and the interrupt is queued by the Power hypervisor. They do not have a a big impact on performance. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
25
lparstat -h A phantom interrupt is an interrupt targeted to another partition that shares the same physical processor. For example, one partition starts an I/O operation. While the partition is waiting for the I/O to complete, it cedes the physical processor to another partition. The I/O operation completes and the controller sends an interrupt to the requesting processor, but as the interrupted partition running is not the intended destination, the partition says “not for me” and the interrupt is queued by the Power hypervisor. They do not have a a big impact on performance. Adds: %hypv – the amount of time spent in the Power hypervisor. hcalls – the number of calls AIX made to the Power hypervisor (per sec) Note: requires root access © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
26
lparstat -H Note: requires root access
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
27
lparstat -i © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
28
Is there a way to see other LPARs?
So far all the instrumentation discussed has been related to the current LPAR Is there a way to see other LPARs? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
29
Is a real time monitor with significant capability
topas - cecdisp topas: Is a real time monitor with significant capability -cecdisp has been added for LPAR support © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
30
topas -cecdisp My first exposure to topas –cecdisp output.
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
31
topas –cecdisp - Another
This one is a lot more interesting. Now we have 3 shared (monitored) LPARs and 1 dedicated monitored. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
32
Understanding the Instrumentation
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
33
Experiment © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
34
stress –c 1 –t 60 & One thread pushes a physical cpu to 100% and we’ve hit 100% of our entitlement. Note: physc is shown in processors, not a percentage. Note the lbusy is only 50%, meaning only 1 (of the 2) logical cpus is being used. Note that SMT mode doesn’t limit the ability to fully consume a physical CPU. App = the available physical processors in the shared pool. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
35
stress –c 2 –t 60 & 2 threads still produces physical CPU = 100% and we’ve still hit 100% of our entitlement. Note the lbusy is now 100%, meaning both logical CPUs are now saturated. SMT is allowing 2 logical CPUs to run. In an uncapped environment, why didn’t it get another physical processor involved? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
36
Let’s give this LPAR another VCPU
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
37
Rerun stress –c 1 –t 60 Note we now have 4 logical CPUs. Entitlement of 1.00 has not changed. But we now have 2 more logical CPUs (smt=On) When we run stress with 1 thread we still get 100% of a physical CPU Entitled capacity is also at 100% which. But note lbusy is now at 25%. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
38
What will happen when we go to 3 threads?
Rerun stress –c 2 –t 60 For the first time we have 2 physical processors saturated. Entitlement has finally exceeded 100%! lbusy is 50% What will happen when we go to 3 threads? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
39
Rerun stress –c 3 –t 60 What’s holding us back?
Still have 2 physical processors saturated. Entitlement still 200. lbusy is now 75. What’s holding us back? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
40
Let’s add yet another VCPU…
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
41
Want to make a prediction?
Test 3 Threads again… Want to make a prediction? Physical consumption goes to 3 physical processors. lbusy is now 50. Note we now have 6 logical CPUs. What will happen when we go to 4 threads? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
42
Stress –c 4 –t 60 Physical consumption and entitlement do not change.
At 3 threads we were getting all the resource that was available. With 4 threads we are simply slicing the same resources across an additional thread, so no more work can actually get done. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
43
Putting it into practice
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
44
Before PowerVM support
harpo AIX 5.3 karl AIX 5.3 marx VIO Server groucho Linux (RedHat) gummo Linux (SuSE) oprah AIX 5.3 System Viewed Start by painting the mental image of dealing with only the traditional OS view. This shows our existing support of an OS image with the PerfMan for Unix agent. We have provided support for AIX from this point of view for a long time, whether a dedicated server or an LPAR. However, there are several limitations to this approach. - can’t tell they are LPARs or what type (dedicated or shared). - VIO servers are really invisible, since no software can be installed there - no hardware view of the entire server © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
45
If this was running on a MicroPartition, what does 100% represent?
CPU utilization If this was running on a MicroPartition, what does 100% represent? For example: Based on the standard information we can glean about CPU utilization on a AIX system, we would see in this demo scenario that we have peaks of 100% activity. Even in an LPARed environment, these numbers are correct. The mystery is: 100% of what? © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
46
Process Utilization Same Question
another example of the limitation of traditional OS image agent © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
47
Before PowerVM Support
harpo AIX 5.3 Karl AIX 5.3 marx VIO Server groucho Linux (RedHat) gummo Linux (SuSE) oprah AIX 5.3 recap for contrast with next slide - actual physical cpu utilization may be distorted - VIO server would be “invisible” since it is a software appliance, IBM does not support installing third party software (e.g. PerfMan agent) there - no physical view of resources (CEC, managed system) © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
48
With PowerVM Support harpo AIX 5.3 karl AIX 5.3 marx VIO Server
Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared oprah AIX 5.3 1 CPU Dedicated Shared Pool by taking advantage of facilities on the platform, PerfMan for UNIX APV support gives a more complete picture of: true CPU utilization by LPAR physical view of shared and dedicated resources A single LPAR’s PerfMan agent can be configured to gather the “cross-partition” information about other LPARs with or without a PerfMan Agent present there some metrics even possible without an agent improved visibility into VIO server still can’t install an agent on VIO server, but some information about it can be gleaned via cross-partition (CEC) view uses xmtopas and/or SSH to HMC © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
49
Demo Environment Linux (RedHat) Linux (SuSE) harpo AIX 5.3 karl
Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared oprah AIX 5.3 1 CPU Dedicated System Viewed Shared Pool © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
50
LPAR Configuration Info
Besides relating this information back to the introductory material, this would be a good place to explain the difference between “managed LPARs” and those that are “unmanaged”. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
51
What can we see about the CEC?
Physical view harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared oprah AIX 5.3 1 CPU Dedicated What can we see about the CEC? Shared Pool as mentioned before, CEC = Central Electronic Complex – IBM-ese for the physical server (LPAR host) © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
52
Physical CPU utilization
The new Physical CPU Utilization chart shows total processor consumption normalized to 100% for both dedicated and shared LPARs. While you can’t see it in this chart, there are 3 physical processors in the shared pool. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
53
Shared pool utilization
The new Shared Processor Utilization chart provides the first clue as to what is really going on in the Shared Processor Pool of the full CEC. On this chart, each physical processor in the shared pool represents 100% of utilization. The capacity line shows there are 3 physical processors in the Shared Processor Pool during this timeframe. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
54
What about LPARs competing for the Shared Pool?
Shared processor pool What about LPARs competing for the Shared Pool? harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared oprah AIX 5.3 1 CPU Dedicated Shared Pool © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
55
Shared pool utilization
Now we see a very different view of harpo’s real CPU usage! Looking closer at shared LPARs, the Top LPARs by % Shared Processor Utilization provides a vivid picture how much of the total shared pool processors are being consumed by LPAR. Note that harpo is often consuming more than 2 physical processors. That certainly is not reflected in the traditional numbers we garner from iostat and ps. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
56
Memory allocation note, actually allocation to the LPAR(s), not consumption by the guest operating system as the chart title might imply © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
57
Context switches source here is from topas, not lparstat
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
58
Can I see Dedicated LPARs?
harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared oprah AIX 5.3 1 CPU Dedicated Shared Pool © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
59
Dedicated CPU utilization
© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
60
What about the LPAR that is the PerfMan managed system?
LPAR details What about the LPAR that is the PerfMan managed system? harpo AIX 5.3 Shared karl AIX 5.3 Shared marx AIX 5.3 VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared oprah AIX 5.3 1 CPU Dedicated Shared Pool © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
61
Physical CPU utilization
In this scenario, the data is being collected from and by “harpo”. Therefore the “This LPAR” folder contains more detail about what is occurring on harpo. The physical cpu utilization chart is particularly useful because it clearly illustrates harpo’s consumption of the shared pool processors compared to its entitlement levels. In an uncapped LPAR it is possible for an LPAR to exceed it’s maximum entitlement and this chart shows harpo doing exactly that. On this chart, 100% represents 1 physical processor. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
62
Virtual CPU This final combination illustrates how the number of virtual CPUs act as another type of capping for shared LPARs. Note on the top chart, the stress tests at 6:30 and 6:40 would not exceed 200%. The bottom chart shows that at that time, there were only 2 virtual CPUs provided to harpo. Another virtual CPU was allocated at 6:45 and subsequent stress tests could exceed the 200% maximum entitlement. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
63
In Summary PowerVM is a sophisticated virtualization platform modeled after zSeries partitioning. Respectable instrumentation is available from an AIX guest, but it requires a knowledgeable consumer. © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
64
System p Logical Partitioning Guide (SA76-0098-00)
References Advanced POWER Virtualization on IBM System p5: Introduction and Configuration (IBM Redbook SG ) IBM System p Advanced POWER Virtualization Best Practices (IBM Redpaper) System p Logical Partitioning Guide (SA ) The APV book is highly recommended © COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.