1. Troubleshooting the Cause for High CPU utilization with J2EE application on UNIX server
PardhaSaradhi D -Senior Technology Architect
Infosys Limited

2. Abstract
There is increasing concern among developers in identifying the relation and cause for high CPU usage with developed Java/J2EE application running on different workloads. This paper defines a methodology how to identify the process id for high CPU utilization on UNIX server with predefined tools with in UNIX server and also helps in identifying the root cause with in J2EE application which is consuming more CPU time with in the available CPU resources.

3. Introduction:
CPU utilization (or process utilization) is the magnitude of usage for which a central processing unit (CPU) is involved in processing logical steps of a program or system. For example, processing on input/output (I/O) operations or method execution, etc. CPU usage allows to quantify the overall business of the system. When the CPU usage is above threshold, let’s say 99%, the logical request fired on the server may experience lag. In such scenario high CPU usage may indicates insufficient processing capability. To meet the requirement, it is necessary to either the CPU capacity needs to be increased or to analyse the process which is CPU intensive and fine tune the program/application associated with the process. Otherwise the user experience may degrade, who is a service requestor of the server.
In the events of multiple processes, the CPU shares the available resources among the processes. If anyone/multiple of the processes demand more resources for processing a logical task/s, the CPU will allocate the resources within its bucket and resulting high CPU usage. Since the CPU resources are shared among all the processes running on the server, the other running process may experience lag in completion of the task. CPU time may increase because of waiting for input or being suspended to allow other programs to run, increased elapsed time which time includes I/O time and all other types of waits incurred by the other operation, etc.

4. Case study: Identifying the CPU usage
Once the user work load is injected on the application server, the CPU usage on the UNIX server, hosting the application server, can be monitored by using the predefined tools available with the UNIX server.
IBM AIX, HP, Solaris, and Linux: the commands can be used to monitor the CPU usage.
• ‘vmstat 1’ – “vmstat” reports information about processes, memory, paging, block IO, traps, and cpu activity. Additional reports give information on a sampling period of length delay. As below.
With the above command, the percentage of CPU usage by user and system process can be easily identified. In order to understand the CPU usage by application server (hosting the J2EE application), it is necessary to ‘grep’ (ps –ef | grep java) for the java process and identify the process id (PID) of the application server.
In case of IBM AIX, Solaris, HP and Linux servers ‘top’ command will list out all the PIDs running on the UNIX server and their resource utilizations. Depending on the version of ‘top’ command, there are different formats as shown below. However the purpose of the command in the current methodology remains same, is to identify the CPU usage.

5. Solaris:
HP-UX
AIX:

6. Executing tprof Command on AIX server
The tprof command generally reports processor usage for each individual programs and also system as a whole. This command is a useful tool which reports the CPU usage of any process ID associated with a Java, C, C++, or FORTRAN program that might be processor-bound. This is helpful in identifying which sections of the program are most heavily using the processor.
It is necessary to execute the below script/commands to find the CPU usage on AIX server and also execute ‘kill -3 <JAVA_PID>’ for java core generation. Otherwise the below script can be executed to generate the required information from the AIX server.
#!/bin/ksh
 print "StatsGather>> aixperf.sh script starting..."
# Set the first command line argument equal to the variable PROBLEMATIC_PID.
PROBLEMATIC_PID=$1
print "StatsGather>> PROBLEMATIC_PID is: $PROBLEMATIC_PID"
print "StatsGather>> Collecting the first javacore..."
kill -3 $PROBLEMATIC_PID
print "StatsGather>> First javacore complete."
tprof -skex sleep 10&
print "StatsGather>> tprof data complete. Output sent to: sleep.prof"
print "StatsGather>> Collecting the second javacore..."
print "StatsGather>> Second javacore complete."
sleep 5
print "StatsGather>> Collecting the third javacore..."

7. In the above output of tprof command, it is evident that “/usr/java6_64/bin/java” process has a 76.70% of CPU utilization. There are 11 child process IDs associated with java process ID –
TID – thread id of each process ID. The Thread IDs associated with each java process is seen to be utilizing 6.5% CPU approximately.
Java core files (Thread dump) generated, will contain the current status/stack trace of threads, which may be processing a piece of code or even they are idle. Each thread in java core file is associated with thread id in hexadecimal format.
Example: First thread id in above figure for java PID is and its equivalent hexadecimal values is BB011F.
The hexadecimal value of the thread id can be used to identify the thread, which costs 6.5% CPU for a database query operation.

8. Case study: Analysing a sample J2EE application on HP-UX UNIX servers
A sample workload has been tested on HP-UX Unix operating system (server with 4 core processor) having a single thread running with in a java instance. A sample service based program is designed to pull – push records between two tables within a database. If no records exist, thread will sleep and notified if new records are ready for processing.
Glance tool:
HP Glance is powerful. It gives a complete picture, highlighting processes that are starved for resources and those using resources. It provides an expert assistance and uses a comprehensive set of rules developed by performance specialists to alert whenever possible.
Command to be executed: /opt/perf/bin/glance -j 25
Analysing the output files
The content of glance output file is as shown below:
==========================================
gbl cpu utilization = %
alive processes =
active processes =
priority queue =
run queue =
context switch rate =
network packet rate =
disk phys io rate =
memory pageout rate =
total mi lost buffers =

9. The gbl cpu utilization shows how much of the central processing unit's time was spent on various activities during the last interval.
The java process which are running at that instance are shown in the glance output along with process ids (pid) and thread ids (ktid). Glance also highlights the CPU percentile of utilization of each java PID with in gbl cpu utilization. It is recommended to generate java thread dumps for the java process id which has high CPU.
In this case, the ‘global cpu utilization’ is 57%, out of which 99% of the CPU has been utilized by a java process id ‘7804’. A set of java thread dumps has been generated for java PID ‘7804’ at a time interval of 5seconds by issuing command ‘kill ’ in the HP UNIX server.
It is observed that ‘Thread-8’ is in runnable state in all generated thread dumps and stack trace is shown as below:
"Thread-8" prio=7 tid=00d2a800 nid=27 lwp_id= runnable [21b40000]
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.sleep(Native Method)
at com.xxxx.xxx.batch.process.InActiveUserThreadManager.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
Observations: The code snippet for the "Thread-8” execution is as shown below, which runs for infinite loop with asleep of 5 milliseconds.
public void run(){
int sleepTime =5;
Thread currentThread = Thread.currentThread();
try{
while (true){
int counter = inActiveUserThreads.size();
for (int j = 0; j < counter; j++){
…... …… }
Thread.sleep(sleepTime);
}catch (InterruptedException interuptExc){
………………….

10. Conclusion
High CPU utilization on production and test environments on UNIX platforms are very common nowadays while working with j2ee applications. Troubleshooting the cause for high CPU utilization with j2ee application on UNIX can be easily carried out with this approach. This paper demonstrates the methodology for identifying the root cause for abnormal CPU utilizations using inbuilt tools from the unix platforms.

11. References & Appendix
The following websites has been referred to gain knowledge on JCE.

12. Author Biography
PardhaSaradhi did masters and a Bachelor of Engineering in Electrical and Electronics from India. He has started perusing his career in IT industry from 2004 and been working with top IT clients all over the globe from last thirteen years in performance engineering & testing domain. Pardha has been analysing performance issues raised by various banking and telecom clients. He has been providing technical solutions for Java based applications with performance improvements. Combining his expertise in Performance engineering and interest in exploring new methodologies to improve the productivity of his team, Pardha has created many frameworks to analyse and identify application bottlenecks quickly.

13. Thank You!!!