1. Application and Performance Management Techniques for J2EE
“Management Built In” for the Adaptive Enterprise
Scott L. Williams HP

2. Agenda Challenges: Testing, Management, & Repair of Applications
Embracing Management Throughout The Application Lifecycle
Management Tools for Operations
Instrumenting Your Applications For The Operational Environment
Demonstration of Application Instrumentation Tools
Call to Action

3. Challenges: Testing, Management, & Repair of Applications

4. The Ideal: Simple, Quick Resolution & Repair
Requirements
Gathering
Design &
Develop
Testing
Staging
Production 1.
Application is “hanging” in production. Use OV to do “root cause” analysis and send results to developers. 2.
Use “root cause” analysis results to narrow down which software contains bug. Fix bug. 3.
Deploy bug fix to staging. Execute regression tests to verify integrity of bug fix. 4.
Release bug fix to production.

5. No Shared “Context”  Difficult For IT/Ops To Correct Problems
the application
Application
Development
IT/Operations
error
occurs
No way to help developers find error
within the application!

6. Examples of common pain points today
Application integration (many technologies, techniques, etc.)
Poor communication (business+dev+operations)
Deployment failures (insufficient information)
Difficulty configuring application security (technology, process)
Maintenance (again, insufficient shared information)

7. Most time and money is dedicated to maintenance in production
Presentation Title
Most time and money is dedicated to maintenance in production
IT Today
Innovation
10%
30%
Consolidation/
Integration
60%
Operation
Source: HP IT department

8. Removing The Barriers To Success
Presentation Title
Removing The Barriers To Success
Free up developer resources
Today: developers often spend too much time supporting their applications when those applications degrade or fail in production
Goal: spend less time supporting applications and more time creating new ones (greater alignment with new business needs)
Free up IT resources
Today: Large numbers of operations staff; manual processes; “archeology” of poorly documented applications
Goal: Well documented apps; automated troubleshooting; better management at all levels (infrastructure, apps, business processes)
Drive complexity out of the software lifecycle
Today: Hidden assumptions, poor documentation, “silos”
Goal: Meta-data (documentation); “management built in” to applications; context shared throughout entire lifecycle (business-development-operations communicate);
Free up developer resources. In most companies today, developers often spend too much
time supporting their applications when those applications degrade or fail in a production
environment. In an adaptive enterprise, developers would spend much less time supporting
applications and much more time creating new applications that support the evolving business
needs of their company.
· Free up IT resources. Often times, IT departments allocate a large number of personnel to
operations tasks. These operations employees are kept busy trying to deploy applications,
discover the root causes of degraded or failing applications, performing ad-hoc or tedious, handtyped
troubleshooting tasks, etc. A great deal of time is wasted trying to understand the structure
of poorly documented applications that have been deployed into IT. The complexities within IT
cause enterprises to spend too much money to discover and remedy problems with their IT assets.
The life of operations personnel in an adaptive enterprise is much more predictable, structured,
and efficient: operations, developers, and business have shared context which leads to tighter
integration and efficiency.
· Drive complexity out of the software lifecycle. Creating and deploying applications
using today’s techniques is often a very cumbersome, error prone, and unpredictable process.
And one of the primary roadblocks that prevent the process from being more efficient is the “toss
it over the wall” mentality that is due to the disconnect between business, application developers,
and IT. The software construction and deployment process in the adaptive enterprise will
leverage shared context – documentation, techniques, and application management information
gathered at runtime – to streamline & simplify the process of bringing new applications and
solutions online.

9. Adaptive Enterprise design principles
Presentation Title
Adaptive Enterprise design principles
Simplification
Simplify applications and systems to reduce complexity and risk. +
Standardization
Standardize the way IT assets are used with common components and processes.
Applied consistently across:
Business processes
Applications
Infrastructure +
The design center of Adaptive Enterprise. Different customers may need or prefer a focus on one over the others, depending on their assessment or pain points. Ultimately all four can and should be improved. Important that solutions address all three levels – infrastructure, applications, and business processes.
Modularity
Improve performance by managing infrastructure components discretely or collectively. +
Integration
Easily manage and modify the environment through a uniform system of relationships.

10. What is “Management for the Adaptive Enterprise?”
Presentation Title
What is “Management for the Adaptive Enterprise?”
Do more with less
Remove barriers to success
Shift IT Investment from maintenance to innovation
Be ahead of the competition
Be competitive or be outsourced
Customer needs evolve slowly, and several principles remain important.
Customers already manage a changing environment in any way they can – not always efficiently or elegantly. Tools and technology only go so far, then it requires a big investment in process and people to do the rest. The result is slow, inefficient, and expensive response to change. Technology can do much more to improve the speed, reliability, and cost of adapting to a changing environment – and relating it to the business.

11. Moving from maintenance to innovation
Presentation Title
Moving from maintenance to innovation
3 years from now
Today
10%
40%
Main idea:
Today more than ever, CIOs have the opportunity to change the way they invest in IT. Through better management, they can flip the ratio for spending heavily on operations to spending heavily on innovation.
Key thoughts:
Most enterprises are spending the biggest part of their budgets on the care and feeding of the IT services that they currently have, crowding out investments for innovation as a result.
Effective management implementations that span people, processes, and technology can yield improvements in productivity, efficiency, reduced downtime, and automation.
These results can be measured. Within HP, we are tracking this in our own plan of record. Also, there are increasingly formal methodologies to measure your potential and actual returns on management investments.
How Bill would present the slide:
Most enterprises are spending the biggest part of their budgets on the care and feeding of the IT services they currently have. Most of that money is spent on personnel and services costs, which unfortunately, when poorly managed, are also very expensive. Infrastructure costs can often be high as well if IT is utilizing the assets it has poorly.
On average, enterprises are spending about 70 percent of their total it budget on this care and feeding. Worse, the ratio between operations and innovation is getting worse for many enterprises, as budgeting for innovation becomes crowded out entirely.
The value proposition behind focused management investments is quite simple. Management, whether delivered as improvements to your organization‘s people, the streamlining of your processes, or making improvements in the technology used to manage IT, can help you invert the ratio of your spending on operations and innovation.
We have found that you can measure this in some specific ways. IT operations efficiency and productivity are things that can be specifically measured. Before a specific management investment is implemented, how many hours were required to bring a new employee into the IT environment? After you apply stream in processes, automation tools, and provide a better skill set to the people providing the services, how many hours do you think it would take? Another area you can look at is the cost of disrupting the business at large. Although most of those costs, usually about 75 percent, are actually seen in the lines of business, not within the IT budget, it still can become extremely expensive. When attention is focused trying to troubleshoot problems, rather than operations, you can see where there's potential for improvement.
HP is aggressively tracking towards this kind of goal internally. In 2002, over 70 percent of HP's IT spend was around applications in infrastructure maintenance. We're currently on track to spend less than 45 percent on operations and maintenance of IT infrastructure and applications by 2006, leaving over half the IT budget available for innovation. And when you're a company like HP, and three and a half percent of more than $80 billion a year‘s run rate is dedicated to IT, shifting a third of your overall budget from operations to innovation means spending $1 billion a year towards helping HP compete more aggressively. That was our motivation for making focused management investments.
We‘re finding this kind of success on a regular basis with our customers as well. Industry analyst firms such as IDC have performed very detailed investment return studies on management software. When evaluating the role of IT productivity and efficiency in contributing to the savings to the IT organization and the enterprise from reduced downtime, we have found that for every dollar of investment made in management, the enterprise and IT organization will receive about $12 in return.
30%
Innovation
60%
20-25%
In large enterprises up to 75 % are personnel or services costs
Hardware, software and infrastructure only 25 %
Consolidation
35-40%
Operation
Source: HP IT department

12. Embracing Management Throughout The Application Lifecycle

13. Shared “Context”  IT/Ops Can Correct Problems
the application
Application
Development
IT/Operations
error
occurs
No way to help developers find error
within the application!
Shared “Context”  IT/Ops Can Correct Problems
the application
Application
Development
IT/Operations
CONTEXT: instrumentation points, performance metrics,
app meta data, etc.
error
occurs
Use “context” to communicate problem
back to application developers

14. Start Here track track monitor monitor control control track monitor

15. Roles:Who Manages Apps in Production?
Application “Owners” (business sponsors)
Customer Management
Renegotiate Contracts
Measure/Monitor Compliance with SLAs
IT/Operations (traditional, platform management)
Ensure health of hardware, OS, some middleware
1st line of defense when problems occur
Application Managers (newer concept)
Ensure health of application containers and custom applications
2nd line of defense. Can often alter/control/repair without needing dev support.
Software Developers
Last line of defense.
When all else fails, they use all runtime context to debug & repair problems

16. Market Observations & Trends – Gartner
Best-of-breed is most popular strategy.
Application development & support are most often responsible for application management.
Custom applications are most need of management
“Gartner believes …that responsibility for application management increasingly resides outside of the traditional IT operations organizations”.
Responsibility for app
App
development
management
IT operations 40
App support 30
LOB % 20 10
outsourcer
Group
other
Gartner “Application Management Poll Reveals New Trends” April 15, 2004

17. Whom HP OpenView Application Management Targets Today
Presentation Title
Whom HP OpenView Application Management Targets Today
CIO
LOB Owner
Biz/IT Liaison
IT Controller
BSM AM
LOB Finance
Bus Proc Owner
App Manager
IT Ops Manager
Infra Svcs Manager
App Dev
App QA
App Support
Network Mgr
Service Desk Mgr
Server Mgr

18. “Enterprise debugger” IDE Architecture Models Application Owners
Software Developer
Tools:
“Enterprise debugger”
IDE
Architecture Models
Application Owners
(business sponsors)
Tools:
“BPM/BAM Dashboards”
CRM/Service Desk
enterprise “coredump”:
call stack traces
performance info
audit trail
business runtime: SLO’s, KPI’s, etc. business processes
Business,
Application & Environment
Context
application runtime:
JMX, WMI
transaction traces,
etc.
operational runtime:
mgmt events, infra. “health”, etc.
Operations/IT
Tools:
Performance Monitoring
Operational Status Monitoring
Infrastru. Troubleshoot Tools
Application Management
Tools:
Application Troubleshooting (JMX info, etc.)
Service Management Tools

19. Management Tools for Operations

20. 3 Pillars of Management Tracking Monitoring Control
observe aspects of a single thread across multiple components.
technologies: logging & auditing
Monitoring
capture runtime and historical events of a particular component, for the purpose of reporting and notifications.
technologies: Real-time statistics (JMX), logging, & auditing
Control
alter the behavior of the managed component directly without recourse to interrupt the operation of the component.
technologies: JMX, custom techniques

21. What is “Tracking?” Measurement of “operational performance”
Combination of JVM profiling & application logging
Often is “durable” recording of what occurred.
Like “frames in a movie”
Some tools could “play back” the recording.
Helpful during testing & staging

22. What is “Monitoring?”
Measurement of “business performance” (SLAs, KPIs)
Measurement of current “operational health” and “business health”
Much “context” (operational & bus) only known by applications: so use application logging & JMX
Some is “durable” recording of what occurred.
Some is transient or “current snapshot”
Management Events & Corrective Triggers

23. What is “Control?” Automated changes or repair of application.
Manual changes or repair to application.
Often in response to management events “fired” during monitoring.
Sometimes modifies operational aspects of applications.
Sometimes modifies business aspects of applications.
Integral to building an adaptive enterprise and adaptive applications.
J2EE Applications often expose JMX MBeans for this purpose.

24. Management Products for “Tracking”
HP OpenView Transaction Analyzer
HP OpenView Internet Services – measures availability (and response-time) by creating synthetic transactions (aka “robots”)
Borland ServerTrace
Actional (web services)
JVM-level Profiling Tools
Etc.

25. Management Products for “Monitoring”
HP OpenView for Operations, Reporting, Graphing
HP Service Desk
HP OpenView Internet Services + OVTA (real end user response times and volume metrics, essential for SLA monitoring -- e.g. response-time < 5 secs AND volume < 10,000 TPS)
BMC
CA Unicenter
Microsoft Operations Manager & Performance Monitor
Actional (web services)
Etc.

26. Management Products for “Control”
HP OpenView for Operations
BMC
CA Unicenter
Microsoft Operations Manager
Etc.

27. Instrumenting Your Applications For The Operational Environment

28. What should you manage?
Think in terms of an application’s “management interface”
Use SLAs, business requirements, & IT/operations needs to help you decide what management to support
Management is a combination of what can be done at the platform (JVM, J2EE Containers, etc.) & what you need to code
You only need to code context that isn’t available from the platform: (business-specific context, operational statistics not gathered by platform, etc.)

29. operational management
business management
Marketing, Analysts, Architects
Customers
contracts with
end users, service
level agreements,
etc.
operational management
Solution Architects, Industry Trends, etc.
starting-point
management
“best practices”
management
requirements
operational management
new requirements
learned over time
through experience
IT Operations

30. determine “dials” and “knobs” for mgmt
management tools
determine “dials” and “knobs” for mgmt
IT Operations
manage…
are specifications for
Management Interface of Solution
management
requirements
communicate…
are specifications for
Solution (business logic, data, etc.)
functional &
systems requirements

31. Management
Interfaces
“HR” App IT
Management
Interfaces
“Call Center” App
Management
Solution(s)

32. Tracking What To Do: Available Technologies & APIs:
Logs and trace for method “entry” and “exit.”
Performance metrics (at JVM-level as well as app container level)
These are like frame-by-frame recordings of program executions. They are durable
Available Technologies & APIs:
JDK 1.4 or later: use standard java logging
JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport )
JVMTI (JVM Tooling Interface) for tracking/monitoring tools

33. Monitoring What To Do: Available Technologies & APIs:
Logs and trace for operational as well as “business” statistics.
Business audit logs for historical business data (store exchanged business documents, business events, etc.)
JMX and WMI for application instrumentation
Performance metrics (at JVM-level as well as app container level)
Available Technologies & APIs:
JDK 1.4 or later: use standard java logging
JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport )
JMX (MBeans) – “custom” MBeans as well as container-provided

34. Control What To Do: Available Technologies & APIs:
react to predictive “triggers” and actually change running servers & code
modify the behavior of servers & code “in flight” without shutdown
solve application problems/issues without always requiring developers
Available Technologies & APIs:
Custom-coded “control point” management objects using JMX (MBeans)
Custom-coded “control points” using: config files, web services, RMI, etc.
Container-level “control points” accessible via JMX, config files, etc.
React to change of state of “control point” objects

35. Best Practices for coding mgmt interfaces
Use JMX for operational and business “health” & as “control points”
Use JMX to provide “rollups” such as:
averages over time
business-specific stats: # of “high value” customers this hour
current operational load on system
current business activity rollup: $ processing right now, etc.
MBeans are for monitoring & “control”: Change their state to control the application.

36. Averages & Roll-Ups Belong Outside of application’s source
Could provide averages over time: “transactions per hour”, “high value customers served today”.
But!, better to put these calculated metrics outside of the application and outside of MBeans:
Management system configurations (OVO templates)
Management “rules” engines, event engines, etc.
Provide simple, atomic data through JMX. Aggregate it externally.

37. Don’t count on “native” JMX Management Events
With JMX you can subscribe to and generate events.
JMX events are “seen” within the JVM. But aren’t sent outside to the management system.
Different management products have their own event mechanisms. And WSDM (Web Services Distributed Management) standard has another.
If you will be in a heterogeneous environment with many technologies, don’t use JMX or WBEM notifications.
Consider not using “native” JVM management events

38. A Very Simple! Standard MBean
Presentation Title
A Very Simple! Standard MBean
public interface MoviePreviewStatsMBean {
public int getPreviewDownloadsCount (); }
public class MoviePreviewStats implements MoviePreviewStatsMBean, java.io.Serializable {
public MoviePreviewStats () {}
public int getPreviewDownloadsCount () {
return m_nPreviewDownloadsCount;
private int m_nPreviewDownloadsCount = 0
protected void resetPreviewDownloadsCount () {
m_nPreviewDownloadsCount = 0;
protected void incrementPreviewDownloadsCount () {
m_nPreviewDownloadsCount++;
Mention what each of these is good for

39. Different “Flavors” of MBeans
Presentation Title
Standard MBeans
Useful when the structure of managed data is well defined and unlikely to change
Named with “MBean” added on the interface
Attributes with getter/setter methods
When to use: When first using MBeans; when the management interface isn’t changing much.
Dynamic MBeans
must implement “DynamicMBean” interface
is like DII in CORBA – your MBean doesn’t need a method signatures that match how the outside world “sees” it
You supply the “MBeanInfo” meta data for the bean
When to use: When you want a level of indirection between the signature of your MBean and the actual implementation; When impl side is dynamic (at runtime as well as design often changing).
Model MBeans
(A subtype of dynamic MBeans)
“RequiredModelMBean” always available and implements the “ModelMBean” interface
The infrastructure implements most of the state caching, methods, etc. you would normally write yourself.
When to use: when you want to configure MBeans from outside of the source code (like from XML file)
Mention what each of these is good for

40. Publishing Your MBeans
If you want the outside world (management systems) to see your MBeans, you “publish” them to an MBeanServer.
Each MBean that is published is like a “pointer” to an actual Java object.
Consumers of MBeans never “touch” the actual MBeans. Instead they ask the MBeanServer to talk to the MBeans.
Some Best Practices:
As often as possible, use the standard JMX API for this (for instance, BEA lets you use an MBean EJB)
Use unique “domain” names for each application.

41. Code Sample for Publishing an MBean
/**
* Create the MBean. Our application will be responsible for
* updating its value throughout the running of the application. */
Employee employeeMBeanRef = new Employee ();
* Access the MBeanServer. */
MBeanServer mbeanServer = MBeanServerFactory.createMBeanServer();
* Create “object name” for Bean. Publish it into the MBeanServer, using
* "HR_domain" as the domain part of the MBean's name attibute. */
ObjectName beanName =
new ObjectName("HR_domain:Name=Employee&type=manager");
mbeanServer.registerMBean( employeeMBeanRef , beanName);

42. Demonstration of Application Instrumentation Tools

43. HP OpenView for Operations
JMX Metric Builder
edit configuration file
SPI Configuration:
user defined metrics
your application
HP OpenView for Operations
MBeans
(the management
interface)
JMX SPI
Container
MBeans
J2EE Container

44. Call To Action
Think about app instrumentation “holistically” rather than as some APIs. (Think about the “management requirements.”)
Work within a development lifecycle that embraces “manageability built in” as fundamental to success
Determine your management requirements and management interfaces up front.
Build your applications to support: tracking, monitoring, & control.
Use technologies such as JMX, web services & SOA to help you create flexible & robust applications for the adaptive enterprise.

45. Questions