Application and Performance Management Techniques for J2EE “Management Built In” for the Adaptive Enterprise Scott L. Williams HP scott.l.williams@hp.com

Agenda Challenges: Testing, Management, & Repair of Applications Embracing Management Throughout The Application Lifecycle Management Tools for Operations Instrumenting Your Applications For The Operational Environment Demonstration of Application Instrumentation Tools Call to Action

Challenges: Testing, Management, & Repair of Applications

The Ideal: Simple, Quick Resolution & Repair Requirements Gathering Design & Develop Testing Staging Production 1. Application is “hanging” in production. Use OV to do “root cause” analysis and send results to developers. 2. Use “root cause” analysis results to narrow down which software contains bug. Fix bug. 3. Deploy bug fix to staging. Execute regression tests to verify integrity of bug fix. 4. Release bug fix to production.

No Shared “Context”  Difficult For IT/Ops To Correct Problems the application Application Development IT/Operations error occurs No way to help developers find error within the application!

Examples of common pain points today Application integration (many technologies, techniques, etc.) Poor communication (business+dev+operations) Deployment failures (insufficient information) Difficulty configuring application security (technology, process) Maintenance (again, insufficient shared information)

Most time and money is dedicated to maintenance in production Presentation Title Most time and money is dedicated to maintenance in production IT Today Innovation 10% 30% Consolidation/ Integration 60% Operation Source: HP IT department

Removing The Barriers To Success Presentation Title Removing The Barriers To Success Free up developer resources Today: developers often spend too much time supporting their applications when those applications degrade or fail in production Goal: spend less time supporting applications and more time creating new ones (greater alignment with new business needs) Free up IT resources Today: Large numbers of operations staff; manual processes; “archeology” of poorly documented applications Goal: Well documented apps; automated troubleshooting; better management at all levels (infrastructure, apps, business processes) Drive complexity out of the software lifecycle Today: Hidden assumptions, poor documentation, “silos” Goal: Meta-data (documentation); “management built in” to applications; context shared throughout entire lifecycle (business-development-operations communicate); Free up developer resources. In most companies today, developers often spend too much time supporting their applications when those applications degrade or fail in a production environment. In an adaptive enterprise, developers would spend much less time supporting applications and much more time creating new applications that support the evolving business needs of their company. · Free up IT resources. Often times, IT departments allocate a large number of personnel to operations tasks. These operations employees are kept busy trying to deploy applications, discover the root causes of degraded or failing applications, performing ad-hoc or tedious, handtyped troubleshooting tasks, etc. A great deal of time is wasted trying to understand the structure of poorly documented applications that have been deployed into IT. The complexities within IT cause enterprises to spend too much money to discover and remedy problems with their IT assets. The life of operations personnel in an adaptive enterprise is much more predictable, structured, and efficient: operations, developers, and business have shared context which leads to tighter integration and efficiency. · Drive complexity out of the software lifecycle. Creating and deploying applications using today’s techniques is often a very cumbersome, error prone, and unpredictable process. And one of the primary roadblocks that prevent the process from being more efficient is the “toss it over the wall” mentality that is due to the disconnect between business, application developers, and IT. The software construction and deployment process in the adaptive enterprise will leverage shared context – documentation, techniques, and application management information gathered at runtime – to streamline & simplify the process of bringing new applications and solutions online.

Adaptive Enterprise design principles Presentation Title Adaptive Enterprise design principles Simplification Simplify applications and systems to reduce complexity and risk. + Standardization Standardize the way IT assets are used with common components and processes. Applied consistently across: Business processes Applications Infrastructure + The design center of Adaptive Enterprise. Different customers may need or prefer a focus on one over the others, depending on their assessment or pain points. Ultimately all four can and should be improved. Important that solutions address all three levels – infrastructure, applications, and business processes. Modularity Improve performance by managing infrastructure components discretely or collectively. + Integration Easily manage and modify the environment through a uniform system of relationships.

What is “Management for the Adaptive Enterprise?” Presentation Title What is “Management for the Adaptive Enterprise?” Do more with less Remove barriers to success Shift IT Investment from maintenance to innovation Be ahead of the competition Be competitive or be outsourced Customer needs evolve slowly, and several principles remain important. Customers already manage a changing environment in any way they can – not always efficiently or elegantly. Tools and technology only go so far, then it requires a big investment in process and people to do the rest. The result is slow, inefficient, and expensive response to change. Technology can do much more to improve the speed, reliability, and cost of adapting to a changing environment – and relating it to the business.

Moving from maintenance to innovation Presentation Title Moving from maintenance to innovation 3 years from now Today 10% 40% Main idea: Today more than ever, CIOs have the opportunity to change the way they invest in IT. Through better management, they can flip the ratio for spending heavily on operations to spending heavily on innovation. Key thoughts: Most enterprises are spending the biggest part of their budgets on the care and feeding of the IT services that they currently have, crowding out investments for innovation as a result. Effective management implementations that span people, processes, and technology can yield improvements in productivity, efficiency, reduced downtime, and automation. These results can be measured. Within HP, we are tracking this in our own plan of record. Also, there are increasingly formal methodologies to measure your potential and actual returns on management investments. How Bill would present the slide: Most enterprises are spending the biggest part of their budgets on the care and feeding of the IT services they currently have. Most of that money is spent on personnel and services costs, which unfortunately, when poorly managed, are also very expensive. Infrastructure costs can often be high as well if IT is utilizing the assets it has poorly. On average, enterprises are spending about 70 percent of their total it budget on this care and feeding. Worse, the ratio between operations and innovation is getting worse for many enterprises, as budgeting for innovation becomes crowded out entirely. The value proposition behind focused management investments is quite simple. Management, whether delivered as improvements to your organization‘s people, the streamlining of your processes, or making improvements in the technology used to manage IT, can help you invert the ratio of your spending on operations and innovation. We have found that you can measure this in some specific ways. IT operations efficiency and productivity are things that can be specifically measured. Before a specific management investment is implemented, how many hours were required to bring a new employee into the IT environment? After you apply stream in processes, automation tools, and provide a better skill set to the people providing the services, how many hours do you think it would take? Another area you can look at is the cost of disrupting the business at large. Although most of those costs, usually about 75 percent, are actually seen in the lines of business, not within the IT budget, it still can become extremely expensive. When attention is focused trying to troubleshoot problems, rather than operations, you can see where there's potential for improvement. HP is aggressively tracking towards this kind of goal internally. In 2002, over 70 percent of HP's IT spend was around applications in infrastructure maintenance. We're currently on track to spend less than 45 percent on operations and maintenance of IT infrastructure and applications by 2006, leaving over half the IT budget available for innovation. And when you're a company like HP, and three and a half percent of more than $80 billion a year‘s run rate is dedicated to IT, shifting a third of your overall budget from operations to innovation means spending $1 billion a year towards helping HP compete more aggressively. That was our motivation for making focused management investments. We‘re finding this kind of success on a regular basis with our customers as well. Industry analyst firms such as IDC have performed very detailed investment return studies on management software. When evaluating the role of IT productivity and efficiency in contributing to the savings to the IT organization and the enterprise from reduced downtime, we have found that for every dollar of investment made in management, the enterprise and IT organization will receive about $12 in return. 30% Innovation 60% 20-25% In large enterprises up to 75 % are personnel or services costs Hardware, software and infrastructure only 25 % Consolidation 35-40% Operation Source: HP IT department

Embracing Management Throughout The Application Lifecycle

Shared “Context”  IT/Ops Can Correct Problems the application Application Development IT/Operations error occurs No way to help developers find error within the application! Shared “Context”  IT/Ops Can Correct Problems the application Application Development IT/Operations CONTEXT: instrumentation points, performance metrics, app meta data, etc. error occurs Use “context” to communicate problem back to application developers

Start Here track track monitor monitor control control track monitor

Roles:Who Manages Apps in Production? Application “Owners” (business sponsors) Customer Management Renegotiate Contracts Measure/Monitor Compliance with SLAs IT/Operations (traditional, platform management) Ensure health of hardware, OS, some middleware 1st line of defense when problems occur Application Managers (newer concept) Ensure health of application containers and custom applications 2nd line of defense. Can often alter/control/repair without needing dev support. Software Developers Last line of defense. When all else fails, they use all runtime context to debug & repair problems

Market Observations & Trends – Gartner Best-of-breed is most popular strategy. Application development & support are most often responsible for application management. Custom applications are most need of management “Gartner believes …that responsibility for application management increasingly resides outside of the traditional IT operations organizations”. Responsibility for app App development management IT operations 40 App support 30 LOB % 20 10 outsourcer Group other Gartner “Application Management Poll Reveals New Trends” April 15, 2004

Whom HP OpenView Application Management Targets Today Presentation Title Whom HP OpenView Application Management Targets Today CIO LOB Owner Biz/IT Liaison IT Controller BSM AM LOB Finance Bus Proc Owner App Manager IT Ops Manager Infra Svcs Manager App Dev App QA App Support Network Mgr Service Desk Mgr Server Mgr

“Enterprise debugger” IDE Architecture Models Application Owners Software Developer Tools: “Enterprise debugger” IDE Architecture Models Application Owners (business sponsors) Tools: “BPM/BAM Dashboards” CRM/Service Desk enterprise “coredump”: call stack traces performance info audit trail business runtime: SLO’s, KPI’s, etc. business processes Business, Application & Environment Context application runtime: JMX, WMI transaction traces, etc. operational runtime: mgmt events, infra. “health”, etc. Operations/IT Tools: Performance Monitoring Operational Status Monitoring Infrastru. Troubleshoot Tools Application Management Tools: Application Troubleshooting (JMX info, etc.) Service Management Tools

Management Tools for Operations

3 Pillars of Management Tracking Monitoring Control observe aspects of a single thread across multiple components. technologies: logging & auditing Monitoring capture runtime and historical events of a particular component, for the purpose of reporting and notifications. technologies: Real-time statistics (JMX), logging, & auditing Control alter the behavior of the managed component directly without recourse to interrupt the operation of the component. technologies: JMX, custom techniques

What is “Tracking?” Measurement of “operational performance” Combination of JVM profiling & application logging Often is “durable” recording of what occurred. Like “frames in a movie” Some tools could “play back” the recording. Helpful during testing & staging

What is “Monitoring?” Measurement of “business performance” (SLAs, KPIs) Measurement of current “operational health” and “business health” Much “context” (operational & bus) only known by applications: so use application logging & JMX Some is “durable” recording of what occurred. Some is transient or “current snapshot” Management Events & Corrective Triggers

What is “Control?” Automated changes or repair of application. Manual changes or repair to application. Often in response to management events “fired” during monitoring. Sometimes modifies operational aspects of applications. Sometimes modifies business aspects of applications. Integral to building an adaptive enterprise and adaptive applications. J2EE Applications often expose JMX MBeans for this purpose.

Management Products for “Tracking” HP OpenView Transaction Analyzer HP OpenView Internet Services – measures availability (and response-time) by creating synthetic transactions (aka “robots”) Borland ServerTrace Actional (web services) JVM-level Profiling Tools Etc.

Management Products for “Monitoring” HP OpenView for Operations, Reporting, Graphing HP Service Desk HP OpenView Internet Services + OVTA (real end user response times and volume metrics, essential for SLA monitoring -- e.g. response-time < 5 secs AND volume < 10,000 TPS) BMC CA Unicenter Microsoft Operations Manager & Performance Monitor Actional (web services) Etc.

Management Products for “Control” HP OpenView for Operations BMC CA Unicenter Microsoft Operations Manager Etc.

Instrumenting Your Applications For The Operational Environment

What should you manage? Think in terms of an application’s “management interface” Use SLAs, business requirements, & IT/operations needs to help you decide what management to support Management is a combination of what can be done at the platform (JVM, J2EE Containers, etc.) & what you need to code You only need to code context that isn’t available from the platform: (business-specific context, operational statistics not gathered by platform, etc.)

operational management business management Marketing, Analysts, Architects Customers contracts with end users, service level agreements, etc. operational management Solution Architects, Industry Trends, etc. starting-point management “best practices” management requirements operational management new requirements learned over time through experience IT Operations

determine “dials” and “knobs” for mgmt management tools determine “dials” and “knobs” for mgmt IT Operations manage… are specifications for Management Interface of Solution management requirements communicate… are specifications for Solution (business logic, data, etc.) functional & systems requirements

Management Interfaces “HR” App IT Management Interfaces “Call Center” App Management Solution(s)

Tracking What To Do: Available Technologies & APIs: Logs and trace for method “entry” and “exit.” Performance metrics (at JVM-level as well as app container level) These are like frame-by-frame recordings of program executions. They are durable Available Technologies & APIs: JDK 1.4 or later: use standard java logging JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport ) JVMTI (JVM Tooling Interface) for tracking/monitoring tools

Monitoring What To Do: Available Technologies & APIs: Logs and trace for operational as well as “business” statistics. Business audit logs for historical business data (store exchanged business documents, business events, etc.) JMX and WMI for application instrumentation Performance metrics (at JVM-level as well as app container level) Available Technologies & APIs: JDK 1.4 or later: use standard java logging JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport ) JMX (MBeans) – “custom” MBeans as well as container-provided

Control What To Do: Available Technologies & APIs: react to predictive “triggers” and actually change running servers & code modify the behavior of servers & code “in flight” without shutdown solve application problems/issues without always requiring developers Available Technologies & APIs: Custom-coded “control point” management objects using JMX (MBeans) Custom-coded “control points” using: config files, web services, RMI, etc. Container-level “control points” accessible via JMX, config files, etc. React to change of state of “control point” objects

Best Practices for coding mgmt interfaces Use JMX for operational and business “health” & as “control points” Use JMX to provide “rollups” such as: averages over time business-specific stats: # of “high value” customers this hour current operational load on system current business activity rollup: $ processing right now, etc. MBeans are for monitoring & “control”: Change their state to control the application.

Averages & Roll-Ups Belong Outside of application’s source Could provide averages over time: “transactions per hour”, “high value customers served today”. But!, better to put these calculated metrics outside of the application and outside of MBeans: Management system configurations (OVO templates) Management “rules” engines, event engines, etc. Provide simple, atomic data through JMX. Aggregate it externally.

Don’t count on “native” JMX Management Events With JMX you can subscribe to and generate events. JMX events are “seen” within the JVM. But aren’t sent outside to the management system. Different management products have their own event mechanisms. And WSDM (Web Services Distributed Management) standard has another. If you will be in a heterogeneous environment with many technologies, don’t use JMX or WBEM notifications. Consider not using “native” JVM management events

A Very Simple! Standard MBean Presentation Title A Very Simple! Standard MBean public interface MoviePreviewStatsMBean { public int getPreviewDownloadsCount (); } public class MoviePreviewStats implements MoviePreviewStatsMBean, java.io.Serializable { public MoviePreviewStats () {} public int getPreviewDownloadsCount () { return m_nPreviewDownloadsCount; private int m_nPreviewDownloadsCount = 0 protected void resetPreviewDownloadsCount () { m_nPreviewDownloadsCount = 0; protected void incrementPreviewDownloadsCount () { m_nPreviewDownloadsCount++; Mention what each of these is good for

Different “Flavors” of MBeans Presentation Title Standard MBeans Useful when the structure of managed data is well defined and unlikely to change Named with “MBean” added on the interface Attributes with getter/setter methods When to use: When first using MBeans; when the management interface isn’t changing much. Dynamic MBeans must implement “DynamicMBean” interface is like DII in CORBA – your MBean doesn’t need a method signatures that match how the outside world “sees” it You supply the “MBeanInfo” meta data for the bean When to use: When you want a level of indirection between the signature of your MBean and the actual implementation; When impl side is dynamic (at runtime as well as design often changing). Model MBeans (A subtype of dynamic MBeans) “RequiredModelMBean” always available and implements the “ModelMBean” interface The infrastructure implements most of the state caching, methods, etc. you would normally write yourself. When to use: when you want to configure MBeans from outside of the source code (like from XML file) Mention what each of these is good for

Publishing Your MBeans If you want the outside world (management systems) to see your MBeans, you “publish” them to an MBeanServer. Each MBean that is published is like a “pointer” to an actual Java object. Consumers of MBeans never “touch” the actual MBeans. Instead they ask the MBeanServer to talk to the MBeans. Some Best Practices: As often as possible, use the standard JMX API for this (for instance, BEA lets you use an MBean EJB) Use unique “domain” names for each application.

Code Sample for Publishing an MBean /** * Create the MBean. Our application will be responsible for * updating its value throughout the running of the application. */ Employee employeeMBeanRef = new Employee (); * Access the MBeanServer. */ MBeanServer mbeanServer = MBeanServerFactory.createMBeanServer(); * Create “object name” for Bean. Publish it into the MBeanServer, using * "HR_domain" as the domain part of the MBean's name attibute. */ ObjectName beanName = new ObjectName("HR_domain:Name=Employee&type=manager"); mbeanServer.registerMBean( employeeMBeanRef , beanName);

Demonstration of Application Instrumentation Tools

HP OpenView for Operations JMX Metric Builder edit configuration file SPI Configuration: user defined metrics your application HP OpenView for Operations MBeans (the management interface) JMX SPI Container MBeans J2EE Container

Call To Action Think about app instrumentation “holistically” rather than as some APIs. (Think about the “management requirements.”) Work within a development lifecycle that embraces “manageability built in” as fundamental to success Determine your management requirements and management interfaces up front. Build your applications to support: tracking, monitoring, & control. Use technologies such as JMX, web services & SOA to help you create flexible & robust applications for the adaptive enterprise.

Questions