Application and Performance Management Techniques for J2EE “Management Built In” for the Adaptive Enterprise Scott L. Williams HP

Agenda Challenges: Testing, Management, & Repair of Applications Embracing Management Throughout The Application Lifecycle Management Tools for Operations Instrumenting Your Applications For The Operational Environment Demonstration of Application Instrumentation Tools Call to Action

Challenges: Testing, Management, & Repair of Applications

Requirements Gathering Design & Develop Testing Staging Production 1. Application is “hanging” in production. Use OV to do “root cause” analysis and send results to developers. 2. Use “root cause” analysis results to narrow down which software contains bug. Fix bug. 3. Deploy bug fix to staging. Execute regression tests to verify integrity of bug fix. 4. Release bug fix to production. The Ideal: Simple, Quick Resolution & Repair

Application Development IT/Operations error occurs No way to help developers find error within the application! the application No Shared “Context”  Difficult For IT/Ops To Correct Problems

Examples of common pain points today Application integration (many technologies, techniques, etc.) Poor communication (business+dev+operations) Deployment failures (insufficient information) Difficulty configuring application security (technology, process) Maintenance (again, insufficient shared information)

Most time and money is dedicated to maintenance in production 10% Operation Consolidation/ Integration Innovation 30% 60% IT Today Source: HP IT department

Removing The Barriers To Success Free up developer resources –Today: developers often spend too much time supporting their applications when those applications degrade or fail in production –Goal: spend less time supporting applications and more time creating new ones (greater alignment with new business needs) Free up IT resources –Today: Large numbers of operations staff; manual processes; “archeology” of poorly documented applications –Goal: Well documented apps; automated troubleshooting; better management at all levels (infrastructure, apps, business processes) Drive complexity out of the software lifecycle –Today: Hidden assumptions, poor documentation, “silos” –Goal: Meta-data (documentation); “management built in” to applications; context shared throughout entire lifecycle (business-development- operations communicate);

Adaptive Enterprise design principles Easily manage and modify the environment through a uniform system of relationships. Improve performance by managing infrastructure components discretely or collectively. Standardize the way IT assets are used with common components and processes. Simplify applications and systems to reduce complexity and risk. Applied consistently across: Business processes Applications Infrastructure Simplification Standardization Modularity Integration

What is “Management for the Adaptive Enterprise?” Do more with less Remove barriers to success Shift IT Investment from maintenance to innovation Be ahead of the competition Be competitive or be outsourced

Moving from maintenance to innovation 10% 40% Operation Consolidation Innovation 30% 60% Today 3 years from now 20-25% 35-40% In large enterprises up to 75 % are personnel or services costs Hardware, software and infrastructure only 25 % Source: HP IT department

Embracing Management Throughout The Application Lifecycle

Application Development IT/Operations error occurs No way to help developers find error within the application! the application Application Development IT/Operations Shared “Context”  IT/Ops Can Correct Problems error occurs Use “context” to communicate problem back to application developers the application CONTEXT: instrumentation points, performance metrics, app meta data, etc.

track monitor control track monitor control track monitor control Start Here

Roles:Who Manages Apps in Production? Application “Owners” (business sponsors) –Customer Management –Renegotiate Contracts –Measure/Monitor Compliance with SLAs IT/Operations (traditional, platform management) –Ensure health of hardware, OS, some middleware –1 st line of defense when problems occur Application Managers (newer concept) –Ensure health of application containers and custom applications –2 nd line of defense. Can often alter/control/repair without needing dev support. Software Developers –Last line of defense. –When all else fails, they use all runtime context to debug & repair problems

Market Observations & Trends – Gartner % Group Responsibility for app management App development IT operations App support LOB outsourcer other Best-of-breed is most popular strategy. Application development & support are most often responsible for application management. Custom applications are most need of management “Gartner believes …that responsibility for application management increasingly resides outside of the traditional IT operations organizations”. Gartner “Application Management Poll Reveals New Trends” April 15, 2004

BSM AM Whom HP OpenView Application Management Targets Today CIO App Manager LOB Owner App Dev App Support IT Ops Manager Network Mgr Service Desk Mgr Biz/IT Liaison Bus Proc Owner IT Controller Infra Svcs Manager Server Mgr App QA LOB Finance

Software Developer Tools: “Enterprise debugger” IDE Architecture Models Application Owners (business sponsors) Tools: “BPM/BAM Dashboards” CRM/Service Desk Operations/IT Tools: Performance Monitoring Operational Status Monitoring Infrastru. Troubleshoot Tools Application Management Tools: Application Troubleshooting (JMX info, etc.) Service Management Tools Business, Application & Environment Context enterprise “coredump”: call stack traces performance info audit trail business runtime: SLO’s, KPI’s, etc. business processes operational runtime: mgmt events, infra. “health”, etc. application runtime: JMX, WMI transaction traces, etc.

Management Tools for Operations

3 Pillars of Management Tracking –observe aspects of a single thread across multiple components. –technologies: logging & auditing Monitoring –capture runtime and historical events of a particular component, for the purpose of reporting and notifications. –technologies: Real-time statistics (JMX), logging, & auditing Control –alter the behavior of the managed component directly without recourse to interrupt the operation of the component. –technologies: JMX, custom techniques

What is “Tracking?” Measurement of “operational performance” Combination of JVM profiling & application logging Often is “durable” recording of what occurred. Like “frames in a movie” Some tools could “play back” the recording. Helpful during testing & staging

What is “Monitoring?” Measurement of “business performance” (SLAs, KPIs) Measurement of current “operational health” and “business health” Much “context” (operational & bus) only known by applications: so use application logging & JMX Some is “durable” recording of what occurred. Some is transient or “current snapshot” Management Events & Corrective Triggers

What is “Control?” Automated changes or repair of application. Manual changes or repair to application. Often in response to management events “fired” during monitoring. Sometimes modifies operational aspects of applications. Sometimes modifies business aspects of applications. Integral to building an adaptive enterprise and adaptive applications. J2EE Applications often expose JMX MBeans for this purpose.

Management Products for “Tracking” HP OpenView Transaction Analyzer HP OpenView Internet Services – measures availability (and response-time) by creating synthetic transactions (aka “robots”) Borland ServerTrace Actional (web services) JVM-level Profiling Tools Etc.

Management Products for “Monitoring” HP OpenView for Operations, Reporting, Graphing HP Service Desk HP OpenView Internet Services + OVTA (real end user response times and volume metrics, essential for SLA monitoring -- e.g. response-time < 5 secs AND volume < 10,000 TPS) BMC CA Unicenter Microsoft Operations Manager & Performance Monitor Actional (web services) Etc.

Management Products for “Control” HP OpenView for Operations BMC CA Unicenter Microsoft Operations Manager Etc.

Instrumenting Your Applications For The Operational Environment

What should you manage? Think in terms of an application’s “management interface” Use SLAs, business requirements, & IT/operations needs to help you decide what management to support Management is a combination of what can be done at the platform (JVM, J2EE Containers, etc.) & what you need to code You only need to code context that isn’t available from the platform: (business-specific context, operational statistics not gathered by platform, etc.)

management requirements starting-point management “best practices” Solution Architects, Industry Trends, etc. operational management new requirements learned over time through experience operational management IT Operations contracts with end users, service level agreements, etc. business management Customers Marketing, Analysts, Architects

Solution (business logic, data, etc.) Management Interface of Solution management tools IT Operations management requirements communicate… are specifications for manage… determine “dials” and “knobs” for mgmt functional & systems requirements

“HR” App Management Interfaces “Call Center” App Management Interfaces IT Management Solution(s)

Tracking What To Do: –Logs and trace for method “entry” and “exit.” –Performance metrics (at JVM-level as well as app container level) –These are like frame-by-frame recordings of program executions. They are durable Available Technologies & APIs: –JDK 1.4 or later: use standard java logging –JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport ) –JVMTI (JVM Tooling Interface) for tracking/monitoring tools

Monitoring What To Do: –Logs and trace for operational as well as “business” statistics. –Business audit logs for historical business data (store exchanged business documents, business events, etc.) –JMX and WMI for application instrumentation –Performance metrics (at JVM-level as well as app container level) Available Technologies & APIs: –JDK 1.4 or later: use standard java logging –JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport ) –JMX (MBeans) – “custom” MBeans as well as container-provided

Control What To Do: –react to predictive “triggers” and actually change running servers & code –modify the behavior of servers & code “in flight” without shutdown –solve application problems/issues without always requiring developers Available Technologies & APIs: –Custom-coded “control point” management objects using JMX (MBeans) –Custom-coded “control points” using: config files, web services, RMI, etc. –Container-level “control points” accessible via JMX, config files, etc. –React to change of state of “control point” objects

Best Practices for coding mgmt interfaces Use JMX for operational and business “health” & as “control points” Use JMX to provide “rollups” such as: –averages over time –business-specific stats: # of “high value” customers this hour –current operational load on system –current business activity rollup: $ processing right now, etc. MBeans are for monitoring & “control”: Change their state to control the application.

Could provide averages over time: “transactions per hour”, “high value customers served today”. But!, better to put these calculated metrics outside of the application and outside of MBeans: –Management system configurations (OVO templates) –Management “rules” engines, event engines, etc. Provide simple, atomic data through JMX. Aggregate it externally. Averages & Roll-Ups Belong Outside of application’s source

Don’t count on “native” JMX Management Events With JMX you can subscribe to and generate events. JMX events are “seen” within the JVM. But aren’t sent outside to the management system. Different management products have their own event mechanisms. And WSDM (Web Services Distributed Management) standard has another. If you will be in a heterogeneous environment with many technologies, don’t use JMX or WBEM notifications. Consider not using “native” JVM management events

public interface MoviePreviewStatsMBean { public int getPreviewDownloadsCount (); } public class MoviePreviewStats implements MoviePreviewStatsMBean, java.io.Serializable { public MoviePreviewStats () {} public int getPreviewDownloadsCount () { return m_nPreviewDownloadsCount; } private int m_nPreviewDownloadsCount = 0 protected void resetPreviewDownloadsCount () { m_nPreviewDownloadsCount = 0; } protected void incrementPreviewDownloadsCount () { m_nPreviewDownloadsCount++; } A Very Simple! Standard MBean

Standard MBeans –Useful when the structure of managed data is well defined and unlikely to change –Named with “MBean” added on the interface –Attributes with getter/setter methods –When to use: When first using MBeans; when the management interface isn’t changing much. Dynamic MBeans –must implement “DynamicMBean” interface –is like DII in CORBA – your MBean doesn’t need a method signatures that match how the outside world “sees” it –You supply the “MBeanInfo” meta data for the bean –When to use: When you want a level of indirection between the signature of your MBean and the actual implementation; When impl side is dynamic (at runtime as well as design often changing). Model MBeans –(A subtype of dynamic MBeans) –“RequiredModelMBean” always available and implements the “ModelMBean” interface –The infrastructure implements most of the state caching, methods, etc. you would normally write yourself. –When to use: when you want to configure MBeans from outside of the source code (like from XML file) Different “Flavors” of MBeans

Publishing Your MBeans If you want the outside world (management systems) to see your MBeans, you “publish” them to an MBeanServer. Each MBean that is published is like a “pointer” to an actual Java object. Consumers of MBeans never “touch” the actual MBeans. Instead they ask the MBeanServer to talk to the MBeans. Some Best Practices: –As often as possible, use the standard JMX API for this (for instance, BEA lets you use an MBean EJB) –Use unique “domain” names for each application.

Code Sample for Publishing an MBean /** * Create the MBean. Our application will be responsible for * updating its value throughout the running of the application. */ Employee employeeMBeanRef = new Employee (); /** * Access the MBeanServer. */ MBeanServer mbeanServer = MBeanServerFactory.createMBeanServer(); /** * Create “object name” for Bean. Publish it into the MBeanServer, using * "HR_domain" as the domain part of the MBean's name attibute. */ ObjectName beanName = new ObjectName("HR_domain:Name=Employee&type=manager"); mbeanServer.registerMBean( employeeMBeanRef, beanName);

Demonstration of Application Instrumentation Tools

J2EE Container your application MBeans (the management interface) HP OpenView for Operations SPI Configuration: user defined metrics JMX SPI JMX Metric Builder edit configuration file Container MBeans

Call To Action Think about app instrumentation “holistically” rather than as some APIs. (Think about the “management requirements.”) Work within a development lifecycle that embraces “manageability built in” as fundamental to success Determine your management requirements and management interfaces up front. Build your applications to support: tracking, monitoring, & control. Use technologies such as JMX, web services & SOA to help you create flexible & robust applications for the adaptive enterprise.

Questions