1. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Exhausting Your Means or Leaving No Stone Unturned Doug Tapscott Systems Analyst VI Siemens Medical Solutions USA, Inc. Malvern, PA

2. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 2 Exhausting Your Means, Leaving No Stone Unturned  Agenda  What’s that mean?  Rules of Thumb  Matrix  Examples

3. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 3 Exhausting Your Means, Leaving No Stone Unturned  What’s that mean?  Exhausting your means equals leaving no turn unstoned.  It means looking in all the areas, not just the performance counters.  It means identifying the Key Performance Indicators (KPI’s).  It means understanding the applications performance profile/behavior with respect to the hosting hardware and O/S (performance trending).  It means understanding the application architecture/environment (what’s running where and why).

4. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 4 Exhausting Your Means, Leaving No Stone Unturned  Rules of Thumb  Charts have more impact than raw numbers, and raw numbers have more impact than opinion.  Let the charts and/or the data tell the story.  Work in specific time periods where possible.  Know the hardware and the O/S.  Know the architecture/environment.  Know the Key Performance Indicators (KPI’s) and the performance profile.  Know what changed most recently.  Know the difference between “Alert Monitoring” and “Performance Trending”.  Understand the tools at your disposal.  Be willing to learn.

5. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 5 Exhausting Your Means, Leaving No Stone Unturned  Matrix We’ve all seen the server trouble shooting and performance management flow chart. With “Yes/No” decision points and “Action” items, it addresses the major areas of:  CPU  Memory  Disk  I/O  Network The focus is trouble shooting. It is difficult to see beyond this very simple decision ladder to the way the Operating system and the application are behaving.

6. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 6 Exhausting Your Means, Leaving No Stone Unturned  Matrix The hardware is the bottom layer. The next is the Operating System. The top is the Application. Hardware Operating System Hardware Operating System Application

7. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 7 Exhausting Your Means, Leaving No Stone Unturned  Matrix  Combine the ladder with the layered view to get the matrix.  Each cell of the matrix may be addressed with:  Performance counters  Various 3 rd party tools (network, SAN Storage, O/S, scripts, etc)  Various native O/S tools

8. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 8 Exhausting Your Means, Leaving No Stone Unturned  Matrix HardwareOperating SystemApplication CPU Memory Storage I/O Network

9. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 9 Exhausting Your Means, Leaving No Stone Unturned  Matrix HardwareOperating SystemApplication CPU # of CPU’s Total CPU% System% User% Process CPU Time Process CPU% Memory Available Memory Used Memory Paged Pool Virtual Mem Managmnt Process Virtual Bytes Working Set Size Storage Phys Transfer time Phys Queue Length Logical Space Available Logical Space Use Delta Process Reads Process Writes I/O Queue Length Wait I/O Bytes/Sec Disk Transfer Rate Process I/O Reads Process I/O Data Network NIC Speed Collisions/Sec TCPIP Avg Bytes/Sec TCPIP Retransmits Process I/O Other Process I/O Other Op/Sec

10. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 10 Exhausting Your Means, Leaving No Stone Unturned  Examples – Scenario 1 An application server needed updates to handle added application features. This included application updates, O/S patches, firmware updates and memory (2GB to 4GB). The server appeared to come up okay. All services started as expected. No alerts were generated. No issues were recorded. When users began to access the application, a response time issue was immediately noted. The first proposal was to back off the patches and the application updates and reboot. The second proposal was that the SAN was at fault. The third proposal was that the network was to blame. Items noted initially: Increased Paging Extended Disk Read Times

11. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 11 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1

12. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 12 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1

13. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 13 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1

14. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 14 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 So far we don’t have the root cause. HardwareOperating SystemApplication CPU Nothing Memory Increased Page File Utilization Storage Increased Disk I/O Increased Disk Read Time I/O Increased Disk I/O Network Nothing

15. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 15 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 What about Memory? Tremendous decrease in Available Memory.

16. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 16 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Did the memory needs increase? No, Committed Bytes haven’t changed.

17. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 17 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Was something suddenly using more memory? No, in fact, every thing is suffering.

18. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 18 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Could the application changes have caused this? No, the key app processes are suffering.

19. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 19 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 What about the paging? Notice the Hard Page Faults that started with the reboot.

20. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 20 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 What type of paging was happening? Heavy read activity was pushing Paging.

21. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 21 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 This makes things more interesting. HardwareOperating SystemApplication CPU Nothing Memory Reduced Available Memory Increased Page File Utilization Greatly reduced working set size Storage Heavy reads mem paging Increased Disk I/O Increased Disk Read Time I/O Increased Disk I/O Heavy reads mem paging Network Nothing

22. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 22 Exhausting Your Means, Leaving No Stone Unturned  Examples – Scenario 1 We looked at the server. The original 2GB RAM was upgraded to 4GB of RAM at the outage. What was supposed to be a 4GB RAM server was now a 512MB RAM server. Fully 7/8 of the RAM chips in the server had failed at the replacement and power up reboot. Having lost 83% of RAM, the server was still able to start all the processes and run. When the load increased, the committed bytes need was fulfilled with paging space and continual reads from the SAN disk where the application was installed. The server was memory thrashing. Taking the server down and replacing the memory chips returned the server to it’s targeted 4GB RAM status. This also restored the response time for the end users of the application, quieting the paging and the disk access. How’d the application performance look after the resolution?

23. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 23 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Overall, with 4GB RAM, Memory usage appeared better.

24. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 24 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Committed Bytes didn’t change throughout the incident. So the application updates had no negative affect on the way memory was used.

25. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 25 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Page file activity was lower with 4GB RAM than it had been with 2GB RAM. This was a predicted outcome of the upgrade to 4GB.

26. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 26 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Memory Page Faults were reduced with 4GB RAM vs the previous 2GB RAM.

27. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 27 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 An interesting effect (increase) was seen regarding System code Resident Bytes with 4GB RAM.

28. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 28 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 A very interesting effect (increase) was seen regarding System Driver Resident Bytes with 4GB RAM. The server doesn’t appear to be paging out/in Driver Resident bytes.

29. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 29 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 The SAN drive M: performance returned to it’s previous level.

30. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 30 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 There was a slight improvement of SAN drive M: Read Time performance with 4GB RAM. This was accompanied by reduced Avg Disk Queue Length.

31. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 31 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 There was a decrease in the size of the process working set overall and for the 2 key processes, “CVi” and “cfi”. This was as expected with the application update.

32. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 32 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 There was also a decrease in the process page file bytes overall and for the 2 key processes, “CVi” and “cfi”. This was as expected with the application update.

33. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 33 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 HardwareOperating SystemApplication CPU Nothing Memory Increased MemoryIncreased Avail Memory Improved Mem Utilization Reduced Page File Use Reduced Working set size Storage Reduced Transfer Time Reduced Read TimeReduced Process Page File bytes I/O Reduced Queue Length Improved Response Time Network Nothing Improved Response Time

34. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 34 Exhausting Your Means, Leaving No Stone Unturned  Examples – Scenario 2 A customer has an ERP application that is having troubles with the heavy update schedule. It’s taking longer than 24 hours to apply updates to the application. When they couple this with their O/S updates and firmware updates, they are losing almost 2 days of time at every monthly update. If they put the updates off for a month, they lose almost 3 days of time. This is an *nix server as a single image (no lpar). They were running a separate *nix based database server. The application is deployed as a 2 tier architecture. The app server had 8GB of RAM, 4 internal disks (mirrored), 4 cpu’s and SAN attached storage for the application. All O/S mount points were on the internal disks.

35. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 35 Exhausting Your Means, Leaving No Stone Unturned  Examples – Scenario 2 The server data looked like this:

36. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 36 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2 HardwareOperating SystemApplication CPU Total 50% Wait I/O 29%Cmplerp 5% Memory Nothing Cmplerp using 4.3 MB Storage Disk busy 95%+ “HI”KB-Write 5652/Sec “HI” (x’s 2) Cmplerp using 4.3 MB I/O Throughput to disk 2826KBPS “HI” (x’s 2) Wait I/O 29% Network Nothing Very long response time

37. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 37 Exhausting Your Means, Leaving No Stone Unturned  Examples – Scenario 2 We mapped the /tmp space to hdisk1 & hdisk2. We reviewed the ERP application to confirm its default use of /tmp space during the update process. We used an O/S environment variable to point the application temp space usage to a file structure on the SAN.

38. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 38 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2

39. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 39 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2

40. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 40 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2 HardwareOperating SystemApplication CPU Total 50% 50% increase in Total CPU Wait I/O 29% 95% Decrease in Wait I/O Cmplerp 5% 74% reduction in elapsed update time Memory Nothing Cmplerp using 4.3 MB Storage Disk busy 95%+ “HI”KB-Write 5652/Sec “HI” (x’s 2) 300% increase in Rd & Wrt/sec Cmplerp using 4.3 MB I/O Throughput to disk 2826KBPS “HI” (x’s 2) Wait I/O 29% 95% Decrease in Wait I/O 200% increase in number of compiles/sec Network Nothing Very long response time 74% reduction in elapsed update time

41. Copyright © 2009 Siemens Medical Solutions USA, Inc. All rights reserved. Page 41 Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2 Most important: Customer saw a 74% reduction in the elapsed update time, from 2 day’s to 12 hours (fits within a single day window). The End…of this cycle. Thank You.