1. Application Performance Monitoring One Approach John Slobodnik April 18, 2006 1:30 p.m. CMG Canada

2. Introduction of Product Suite ServerVantage ServerVantage ApplicationVantage ApplicationVantage ClientVantage ClientVantage VantageAnalyzer VantageAnalyzer VantageView VantageView

3. ServerVantage(SV) Collects “server” level data. Collects “server” level data. –Multiplatform: Windows, LINUX, UNIX, etc. –CPU, memory, disk, network out of the box. Collects “application” level data. Collects “application” level data. –Oracle, SQL server, WebLogic, IPlanet, LDAP, etc. One SV agent installed on each client server. One SV agent installed on each client server. –Runs most of the time. Customized counters (metrics) can be written. Customized counters (metrics) can be written.

4. ApplicationVantage (AV) A sniffer. A sniffer. –Agent-based application analysis of packet level communications. Gathers all traffic that passes through the Network Interface Cards (NICs). Gathers all traffic that passes through the Network Interface Cards (NICs). Can merge the data together from multiple servers. Can merge the data together from multiple servers. –Can trace, for example, SQL server traffic. One AV agent is installed per client server. One AV agent is installed per client server. –Turned on when required. –Most often in firefighting mode.

5. ClientVantage (CV) Gathers data on the performance of your application. Gathers data on the performance of your application. –Done through timings of synthetic business transactions on CV workstations (robots). Scripting of business transactions done with a tool called QARun. Scripting of business transactions done with a tool called QARun. –We are doing active monitoring. –There are two other options available here now: Passive monitoring using CV Passive monitoring using CV A hardware-based solution A hardware-based solution

6. Vantage Analyzer for J2EE (VA) J2EE (Java) based tool to help pinpoint exact locations of code-level performance problems. J2EE (Java) based tool to help pinpoint exact locations of code-level performance problems. –Locates slow methods, SQL statements and transactions. The VA agent runs inside your Application servers JVM obtaining performance metrics using Byte Code Instrumentation. The VA agent runs inside your Application servers JVM obtaining performance metrics using Byte Code Instrumentation. –Data sent in real-time to nucleus server where it is stored and distributed to VA performance consoles. Supports WebLogic, Websphere, etc. Supports WebLogic, Websphere, etc. Customized component Customized component –Allows a transaction to be followed in VantageView.

7. VantageView (VV) Web-based portal for viewing SV, CV, AV and VA data for monitoring and reporting. Web-based portal for viewing SV, CV, AV and VA data for monitoring and reporting. Accessing information from the Vantage suite of tools, VantageView users check the status of clients, servers and networks from their intranet while also providing a near-time service level perspective on application availability and performance. Accessing information from the Vantage suite of tools, VantageView users check the status of clients, servers and networks from their intranet while also providing a near-time service level perspective on application availability and performance. The flexibility of VantageView enables different levels of users to view pertinent information for easy problem determination and resolution. The flexibility of VantageView enables different levels of users to view pertinent information for easy problem determination and resolution. Customized counters (metrics) can be created in the VV database. Customized counters (metrics) can be created in the VV database.

9. A Few Easy Setup Steps A summary of the steps to implement the solution: –Install the agents. –Complete Administration –Set Preferences –Management Create tasks and apply blackout schedules. –Create monitoring views. –Create reports. Optional steps taken –Create dashboards. –Create custom counters (metrics).

10. Install the Agents  This is a quick, procedural task that is quick to complete. –A script is run to do the install followed by any applicable patches.  The product keeps track of the level of agent installed on each server in a central repository.  SV and AV agents are installed on each server (Window, UNIX, LINUX)

11. Administration – Configure Databases Set up the online database(s).

12. Configure Historical Database Define the historical database. We keep 3 months of data online. All else goes to historical database.

13. Control Server Configuration Set up the control servers.

14. Define Users Define VV user profiles.

15. Preferences Business applications 4 applications. Business locations Various Canadian cities. Business transactions An application (29 transactions) is broken down into 3 transaction groupings (14, 7, 8). Server groupings Production, pre- production, support, third-party, etc.

16. Management - Create Tasks Create a new task.

17. Create Tasks Select the type of server: Windows, UNIX, etc.

18. Create Tasks Select the counters you wish to see.

19. Create Tasks Add a rule for alerting.

20. Create Tasks Set up alerts if you want them. For example: System Thrashing, TCP Connectivity lost from WL to WL layer, CPU > 90%, etc.

21. Create Tasks If you can do it from a command line it can be automated here: shell scripts, bat files. Perform an action based on a threshold being breached. (1) Kick off a WL thread dump based on a WL counter below a certain level. (2) Send an alert based on an ASCII pattern match. (3) Previous problems can be proactively addressed with this type of instrumentation. We examine WebLogic logs Alert notification via pager, email, SNMP, etc. Different audiences for different tasks, DBAs, App. Support, etc.

22. Create Tasks Select the appropriate data sampling interval. Key to the size of your database.

23. Create Tasks Start the task.

24. Management – Blackout Schedule Apply a blackout schedule, if applicable. Apply a blackout schedule, if applicable. –ServerVantage agents do not run when the application is down daily. Client Vantage robots are also set up to run on a blackout schedule. Client Vantage robots are also set up to run on a blackout schedule. –Implemented through CV which uses the Windows scheduler.

25. Create Monitoring Views  Monitoring views contain all data points.  Flexible: you can plot many different metrics on the same chart.  Business metric vs. server performance.  Application metrics vs. server metrics.  TeeChart Editor gives you Excel chart type functionality to modify the look of the chart.

26. Monitoring View Saved as a permanent monitoring view report.

27. Monitoring View

28. Monitoring (ad hoc) Can drill into data point.

29. Drill into IDP Intelligent Data Point (IDP)

30. Create Reports Reports contain different levels of data summarization. From all data points to daily average. We have created 12 hour, 2 day, 1 week and 1 month views of all reports. Flexible: you can plot many different metrics on the same chart. TeeChart Editor gives you Excel chart type functionality to modify the look of the chart.

31. Create Reports Select the metric source.

32. Create Reports Select the metric(s) desired.

33. Create Reports Select the time range.

34. Create Reports Select the display format.

35. Create Reports Schedule the report.

36. Create Reports Save the report.

37. Reports

40. Then the Business asked… How can we prove that the API calls are performing better? Custom program installed on WL servers. Custom program installed on WL servers. Gathers API call response time data, converts it to a local CSV file, FTP to VV database. Gathers API call response time data, converts it to a local CSV file, FTP to VV database. API Response Time report created, queries VV DB. API Response Time report created, queries VV DB. APIs split between internal vs. outsourced (for reporting purposes). APIs split between internal vs. outsourced (for reporting purposes). There are a number of activities within each bean conversation. There are a number of activities within each bean conversation.

41. API Response Time Report Sample bean conversation report.

42. Then Management said… We need to have a some different dashboard views. We need to have a some different dashboard views. Each level of dashboard gets more detailed. Each level of dashboard gets more detailed. Special dashboard for outsourced infrastructure. Special dashboard for outsourced infrastructure. Dashboards were created using the integrated VISIO (Vantage Visualizer) piece of the product. Dashboards were created using the integrated VISIO (Vantage Visualizer) piece of the product.

43. Management Dashboard

44. Drill down to Application Availability

45. Application Availability (bottom)

46. Drill down to Heat Chart

47. Drill down to CNS report

48. Drill down to Application Scorecard

49. Application Scorecard (bottom)

50. Drill down to Transaction Scorecard

51. Drill down to Performance Summary

52. Drill down to Orders Report

53. Drill down to Session Current Count report

54. Drill down to WL Serviced Requests report

56. Geographic Dashboard

57. We asked ourselves… How can we make this easier to support? How can we make this easier to support? Customized metrics can be created in VV or SV. Customized metrics can be created in VV or SV. Make non-standard types of metrics available. Make non-standard types of metrics available. Samples of some of the customization created: Samples of some of the customization created: Disk usage of SV logs files directory. Disk usage of SV logs files directory. Automated removal of SV log files. Automated removal of SV log files. Automate push of patches to all agents. Automate push of patches to all agents. Send a command to run on a server and return the result. Send a command to run on a server and return the result. Count the number of SV datafiles. Count the number of SV datafiles. Agent restart. Agent restart. Gather SV log files. Gather SV log files.

58. More Customization TCP Connection test from WL layer to WL layer. TCP Connection test from WL layer to WL layer. Number of Orders. Number of Orders. SQL query to xml to csv to VV DB. SQL query to xml to csv to VV DB. ASCII file pattern match in WL logs (3). ASCII file pattern match in WL logs (3). Automatic thread dumps WebLogic. Automatic thread dumps WebLogic. Average Elapsed Time Average Elapsed Time Customer purchase at store experience. Customer purchase at store experience. Individual transaction timings is CV, adding them up is custom. Individual transaction timings is CV, adding them up is custom. Network Test / TCP Connection Test Network Test / TCP Connection Test Traceroute response time for up to 10 hops & alert. Traceroute response time for up to 10 hops & alert. API Response time monitoring. API Response time monitoring. Average, max, min, std dev Average, max, min, std dev

59. Network Connection Test

60. Vantage Analyzer Installed on production WebLogic servers during the peak annual sales period. Now in the pre-prod environment. –So bugs can be found before promoting new code to production.

61. J2EE JavaScape Paints a landscape view of your J2EE environment. This view displays component interactions between JSPs, Servlets and Web services, Session, Entity and Message-driven Beans, as well as database usage.

62. Transaction Explorer The tree is organized by the largest consumers, from top to bottom. The tree can be sorted by the CPU or Transaction time period.

63. Transaction Scope Gives a detailed view on each individual transaction which runs through your application. Stalled Threads Shows thread-level detail of a transaction.

64. Method HotSpots Identifies the biggest consumers in your application. The view can be sorted by Transaction or CPU time.

65. SQLyzer HotSpots Lets you pinpoint the largest SQL consumers. SLA Monitoring This view displays pre-configured SLA rules and when they were last violated.

66. Memory HotSpots Locate memory leaks as well as memory allocation hot spots to help assist with server availability and performance.

67. Summary Management extremely pleased. Customized dashboards, peak period success, want more application’s instrumented. Business application ran almost 99.9% availability during peak processing period of the year in large part due to this solution. Now instrumented to be more proactive than in the past. Being used as a model for the rest of the enterprise. Support teams have embraced the solution because it makes their lives easier. DBA’s, application support, system administrator’s, performance and capacity planners, etc. Significantly less time wasted determining whose problem it is (you know, 6 teams in a room…) during fire-fighting.

68. John.Slobodnik@bell.ca (905) 282-3342