1. Bill Au CBS Interactive http://www.apache.org/~billa/oscon2012/ Troubleshooting Slow or Hung Java Applications

2. 2 Agenda » Tools of the Trade » Case Studies 2

3. 3 Tools – Out Of The Box » jinfo, jmap, jstack, jstat » JConsole, VisualVM –Monitor CPU usage by using JTop plug-in ($JAVA_HOME/demo/management/JTop) » java.lang.management –Platform MXBeans: runtime, operating system, class loading, compilation, garbage collection, memory, thread –Sample code in $JAVA_HOME/demo/management –JDK demo and sample was part of JDK download for jdk6u31 or older, separate download for u32+ 3

4. 4 Tools – java.lang.management » Sample code: import java.lang.management.*; ThreadMXBean tb=ManagementFactory.getThreadMXBean(); if (tb.isThreadCpuTimeSupported()) { long[] tids=tb.getAllThreadIds(); for (long tid : tids) { System.out.println(“cputime:”+tb.getThreadCpuTime(tid)); System.out.println(“usertime:”+tb.getThreadUserTime(tid)); } 4

5. 5 Tools » Garbage collection log –-Xloggc: –Timestamp and duration of GC events –Sizes before and after GC events –More options to explore: −-XX:+PrintGCDateStamps −-XX:+PrintGCDetails −-XX:+PrintTenuringDistribution −-XX:+PrintGCApplicationsStoppedTime –Some GC flags can be set dynamically by jinfo/JConsole/VisualVM: PrintGC, PrintGCDetails, PrintGCTimeStamps » visualgc – GUI for garbage collection monitoring 5

6. 6 Tools » Thread dump –kill -3 (or kill –SIGHUP) –JConsole, VisualVM –ThreadMXBean –jstack (-F) » Consecutive thread dumps –Stop motion animation 6

7. 7 Hung or Slow App » Long GC pauses and/or high GC overhead –Basic GC tuning −max heap size: -Xmx −type of collector used −-XX:+UserSerialGC −-XX:+UseParallelGC/-XX:+UseParallelOldGC −-XX:+UseConcMarkSweepGC −-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC (jdk6u14+) –Advanced GC tuning −young generation size: -Xmn –Baseline, tune, measure one change at a time » Demo: HPjmeter – garbage collection analysis (and more…) 7

8. 8 Hung or Slow App » Deadlock –Take thread dump to run the JVM’s deadlock detector » Looping Threads –Monitor CPU times of running threads −ThreadMXBean −JTop plug-in for JConsole/VisualVM –Inspect stack trace of suspicious threads 8

9. 9 Slow or Hung App » Blocked threads –Look for blocked threads waiting on the same condition –Identify bottleneck for resource contention –Inspect code for incautious and/or improper synchronization » Demo: samurai – thread dump analysis tool » Stuck threads –Look for runnable threads with the same stack trace across consecutive thread dumps –Some common causes: −Blocking for a network call without specifying connect and/or read timeout (default may be no timeout) −Long execution time for calls to backend servers 9

10. 10 Links » Presentation http://www.apache.org/~billa/oscon2012 » HPjmeter https://h20392.www2.hp.com/portal/swdepot/display ProductInfo.do?productNumber=HPJMETER » samurai http://yusuke.homeip.net/samurai/en/index.html https://github.com/yusuke/samurai 10