Bill Au CBS Interactive Troubleshooting Slow or Hung Java Applications

2 Agenda » Tools of the Trade » Case Studies 2

3 Tools – Out Of The Box » jinfo, jmap, jstack, jstat » JConsole, VisualVM –Monitor CPU usage by using JTop plug-in ($JAVA_HOME/demo/management/JTop) » java.lang.management –Platform MXBeans: runtime, operating system, class loading, compilation, garbage collection, memory, thread –Sample code in $JAVA_HOME/demo/management –JDK demo and sample was part of JDK download for jdk6u31 or older, separate download for u32+ 3

4 Tools – java.lang.management » Sample code: import java.lang.management.*; ThreadMXBean tb=ManagementFactory.getThreadMXBean(); if (tb.isThreadCpuTimeSupported()) { long[] tids=tb.getAllThreadIds(); for (long tid : tids) { System.out.println(“cputime:”+tb.getThreadCpuTime(tid)); System.out.println(“usertime:”+tb.getThreadUserTime(tid)); } 4

5 Tools » Garbage collection log –-Xloggc: –Timestamp and duration of GC events –Sizes before and after GC events –More options to explore: −-XX:+PrintGCDateStamps −-XX:+PrintGCDetails −-XX:+PrintTenuringDistribution −-XX:+PrintGCApplicationsStoppedTime –Some GC flags can be set dynamically by jinfo/JConsole/VisualVM: PrintGC, PrintGCDetails, PrintGCTimeStamps » visualgc – GUI for garbage collection monitoring 5

6 Tools » Thread dump –kill -3 (or kill –SIGHUP) –JConsole, VisualVM –ThreadMXBean –jstack (-F) » Consecutive thread dumps –Stop motion animation 6

7 Hung or Slow App » Long GC pauses and/or high GC overhead –Basic GC tuning −max heap size: -Xmx −type of collector used −-XX:+UserSerialGC −-XX:+UseParallelGC/-XX:+UseParallelOldGC −-XX:+UseConcMarkSweepGC −-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC (jdk6u14+) –Advanced GC tuning −young generation size: -Xmn –Baseline, tune, measure one change at a time » Demo: HPjmeter – garbage collection analysis (and more…) 7

8 Hung or Slow App » Deadlock –Take thread dump to run the JVM’s deadlock detector » Looping Threads –Monitor CPU times of running threads −ThreadMXBean −JTop plug-in for JConsole/VisualVM –Inspect stack trace of suspicious threads 8

9 Slow or Hung App » Blocked threads –Look for blocked threads waiting on the same condition –Identify bottleneck for resource contention –Inspect code for incautious and/or improper synchronization » Demo: samurai – thread dump analysis tool » Stuck threads –Look for runnable threads with the same stack trace across consecutive thread dumps –Some common causes: −Blocking for a network call without specifying connect and/or read timeout (default may be no timeout) −Long execution time for calls to backend servers 9

10 Links » Presentation » HPjmeter ProductInfo.do?productNumber=HPJMETER » samurai