Java Performance Analysis 301 Peter Johnson CMG 2006, December 7, Session 534, Paper 6033

Java Performance Analysis 301 Page 2 Dec. 7, 2006 Abstract The Java Platform provides a variety of mechanisms for monitoring the performance of Java applications. There are several tools that are freely available that can be used to monitor Java applications. This paper describes some of those tools, and the Java Management Extensions (JMX) technology on which those tools are built. Additionally, the paper shows how custom tools can be built using JMX.

Java Performance Analysis 301 Page 3 Dec. 7, 2006 Outline Review 101 – garbage collection analysis 201 – garbage collection variations JVM monitoring tools: JConsole JStat Java Management Extensions (JMX) Monitoring an application server (JBoss)

Java Performance Analysis 301 Page 4 Dec. 7, 2006 Java Heap Young Generation Eden Where new objects are created Survivor spaces Where garbage collector places objects that are still in use Old Generation Where tenured objects are placed References

Java Performance Analysis 301 Page 5 Dec. 7, 2006 Minor Garbage Collection Occurs when no room in eden for new object Marks all reachable objects in eden and “from” survivor space Copies those objects to “to” survivor space Updates references accordingly If “to” survivor space fills, overflows to old generation Resets allocation pointer to start of eden Old GenerationEden Spaces AB EF C G D HI To From AB EF C G D HI AB EF C G D HI E G H I E G H I From To

Java Performance Analysis 301 Page 6 Dec. 7, 2006 Major Garbage Collection Occurs when insufficient room in old generation to hold to-be- tenured objects during a minor collection Mark and Compact Reachable objects are marked Marked objects are moved to one end of the old generation Old GenerationEden Spaces To From AB EF C G D HI JK PQ L R M S NO TU AB EF C G D HI JK PQ L R M S NO TU VW X AB EF C G D HI JK PQ MO X

Java Performance Analysis 301 Page 7 Dec. 7, 2006 Garbage Collection Info Options -verbose:gc Heap usage before and after GC Time spent during GC -XX:+PrintGCDetails Generation and heap size before and after GC Time spent during GC -XX:+PrintHeapAtGC Generation sizes before and after GC Space sizes and percent in use before and after GC Memory locations of the heap, its generations, and their spaces

Java Performance Analysis 301 Page 8 Dec. 7, 2006 Gathering GC Data... [GC 1860K->1388K(1984K), secs] [GC 1900K->1446K(1984K), secs] [GC 1958K->1468K(2112K), secs] [Full GC 1468K-> 195K(2112K), secs]... Analyzer ,1388, ,1446, ,1468, ,195, Comma-separated value (CSV) file GC data printed to stdout Simple program w/ reg. expr. >j ava -verbose:gc my.app.Main Spreadsheet

Java Performance Analysis 301 Page 9 Dec. 7, 2006 Interpreting Graphed GC Data Typical GC data graph For many GCs, the heap slowly fills as objects get moved to the old generation Blue (size before GC) or magenta (size after GC) lines with positive slope Then a full GC happens Drop in blue (or magenta) lines Yellow (GC time) dot in higher position

Java Performance Analysis 301 Page 10 Dec. 7, 2006 Parallel and Concurrent Collectors Parallel collector – young generation only -XX:+UseParallelGC -XX:ParallelGCThreads=number By default, number of threads = number of CPUs JVM 5.0 – on by default if >= 2CPUs and >= 1GB RAM Concurrent Mark Sweep collector – tenured gen only -XX:+UseConcMarkSweepGC -XX:+UseParNewGC (parallel young gen collector) Concurrent collection runs in phases Some phases stop the application threads Some phases run concurrently with application threads

Java Performance Analysis 301 Page 11 Dec. 7, 2006 Concurrent Collector Phases Application threads Concurrent GC thread time Initial marking phase Locates all objects directly referenced by a root local variable static field

Java Performance Analysis 301 Page 12 Dec. 7, 2006 Concurrent Collector Phases Application threads Concurrent GC thread time Initial marking phase Locates and marks all objects referenced All new objects marked All changed objects noted Concurrent marking phase

Java Performance Analysis 301 Page 13 Dec. 7, 2006 Concurrent Collector Phases Application threads Concurrent GC thread time Initial marking phase Rechecks all changed objects When done, all live objects are marked Concurrent marking phase Final marking phase

Java Performance Analysis 301 Page 14 Dec. 7, 2006 Concurrent Collector Phases Application threads Concurrent GC thread time Initial marking phase Frees memory used by dead objects Maintains list of available memory Concurrent marking phase Final marking phase Concurrent sweeping phase

Java Performance Analysis 301 Page 15 Dec. 7, 2006 Concurrent Collector Phases Application threads Concurrent GC thread time Initial marking phase Post-collection clean up Concurrent marking phase Final marking phase Concurrent sweeping phase Reset phase

Java Performance Analysis 301 Page 16 Dec. 7, 2006 Outline Review 101 – garbage collection analysis 201 – garbage collection variations JVM monitoring tools: JConsole JStat Java Management Extensions (JMX) Monitoring an application server (JBoss)

Java Performance Analysis 301 Page 17 Dec. 7, 2006 JConsole JVM 5.0 Monitors: Threads Heap usage Classes MBeans

Java Performance Analysis 301 Page 18 Dec. 7, 2006 JConsole JVM 5.0 Monitors: Threads Heap usage Classes MBeans

Java Performance Analysis 301 Page 19 Dec. 7, 2006 JConsole JVM 5.0 Monitors: Threads Heap usage Classes MBeans

Java Performance Analysis 301 Page 20 Dec. 7, 2006 JConsole JVM 5.0 Monitors: Threads Heap usage Classes MBeans

Java Performance Analysis 301 Page 21 Dec. 7, 2006 JConsole JVM 5.0 Monitors: Threads Heap usage Classes MBeans

Java Performance Analysis 301 Page 22 Dec. 7, 2006 JConsole – Memory Tab

Java Performance Analysis 301 Page 23 Dec. 7, 2006 JConsole – Memory Tab Blue line outlines selected heap

Java Performance Analysis 301 Page 24 Dec. 7, 2006 JConsole – Memory Tab

Java Performance Analysis 301 Page 25 Dec. 7, 2006 JConsole – Thread Tab

Java Performance Analysis 301 Page 26 Dec. 7, 2006 JConsole – Classes Tab

Java Performance Analysis 301 Page 27 Dec. 7, 2006 JConsole – MBeans Tab

Java Performance Analysis 301 Page 28 Dec. 7, 2006 JConsole – MBeans Tab Blue values are editable Double-click to expand bold values

Java Performance Analysis 301 Page 29 Dec. 7, 2006 JConsole – MBeans Tab

Java Performance Analysis 301 Page 30 Dec. 7, 2006 JConsole – MBeans Tab

Java Performance Analysis 301 Page 31 Dec. 7, 2006 JConsole – MBeans Tab

Java Performance Analysis 301 Page 32 Dec. 7, 2006 JConsole – VM Tab

Java Performance Analysis 301 Page 33 Dec. 7, 2006 Running the JConsole Two steps: 1: Start Java application with these options: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port= -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false You can also monitor using a secure connection Required, only option needed to monitor local application Use all four options to monitor a remote application. Note that an unsecure connection is used.

Java Performance Analysis 301 Page 34 Dec. 7, 2006 Running the JConsole 2: Start JConsole and connect to the JVM Started JBoss with all four options, with port Can connect using either method. service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi Can supply connection options on JConsole command line.

Java Performance Analysis 301 Page 35 Dec. 7, 2006 The jstat utility Command line utility that provides statistics for Garbage collection Just-in-time compiler Class loading/unloading jstat statistic option process [wait count] Which statistic to report Report output option Java application process id Number of times to get statistics and time to wait between retrievals

Java Performance Analysis 301 Page 36 Dec. 7, 2006 Statistic and Output Options Heap statistics -gc, -gccapacity, -gcnew, -gcutil, and many more... Compiler statistics -compiler, -printcompilation Class loader statistics -class Output options -hn - display column header every n lines -t - display time since JVM started in column 1

Java Performance Analysis 301 Page 37 Dec. 7, 2006 Jstat Output jstat –gcutil –t s 200 Timestamp S0 S1 E O P YGC YGCT FGC FGCT GCT Seconds since start of app Survivor space usage % Eden usage % Tenured gen usage % Permanent space usage % # minor/full collections Time spent in collections Process id Repeat 200 times, once a second

Java Performance Analysis 301 Page 38 Dec. 7, 2006 Outline Review 101 – garbage collection analysis 201 – garbage collection variations JVM monitoring tools: JConsole JStat Java Management Extensions (JMX) Managed Bean (MBean) Monitoring an application server (JBoss)

Java Performance Analysis 301 Page 39 Dec. 7, 2006 Java Management Extensions (JMX) Standard specification for management info Statistics, operations, events Java Specification Request 3 (JSR-3) Server JMXConnector MBeanServer- Connection Client MBean Uses connection to interact with MBeans Ask server for for connector which provides an MBean connection Provide management capabilities You can write your own MBeans

Java Performance Analysis 301 Page 40 Dec. 7, 2006 JMX Example Program public static void main(String[] args) throws Exception { JMXServiceURL url = new JMXServiceURL( "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi"); JMXConnector jconn = JMXConnectorFactory.connect(url); MBeanServerConnection mconn = jconn.getMBeanServerConnection(); ObjectName name = new ObjectName ("java.lang:type=GarbageCollector,name=Copy"); Object val = mconn.getAttribute(name,"CollectionCount"); System.out.println(name + "\n\tCollectionCount=" + val); } This example queries an MBean provided by the JVM.

Java Performance Analysis 301 Page 41 Dec. 7, 2006 JMX Example Program public static void main(String[] args) throws Exception { JMXServiceURL url = new JMXServiceURL( "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi"); JMXConnector jconn = JMXConnectorFactory.connect(url); MBeanServerConnection mconn = jconn.getMBeanServerConnection(); ObjectName name = new ObjectName ("java.lang:type=GarbageCollector,name=Copy"); Object val = mconn.getAttribute(name,"CollectionCount"); System.out.println(name + "\n\tCollectionCount=" + val); } This example queries an MBean provided by the JVM. URL used to connect to the server. Port 9999 matches earlier JConsole example.

Java Performance Analysis 301 Page 42 Dec. 7, 2006 JMX Example Program public static void main(String[] args) throws Exception { JMXServiceURL url = new JMXServiceURL( "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi"); JMXConnector jconn = JMXConnectorFactory.connect(url); MBeanServerConnection mconn = jconn.getMBeanServerConnection(); ObjectName name = new ObjectName ("java.lang:type=GarbageCollector,name=Copy"); Object val = mconn.getAttribute(name,"CollectionCount"); System.out.println(name + "\n\tCollectionCount=" + val); } This example queries an MBean provided by the JVM. Get the JMX connector. For there get the MbeanServerConnection.

Java Performance Analysis 301 Page 43 Dec. 7, 2006 JMX Example Program public static void main(String[] args) throws Exception { JMXServiceURL url = new JMXServiceURL( "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi"); JMXConnector jconn = JMXConnectorFactory.connect(url); MBeanServerConnection mconn = jconn.getMBeanServerConnection(); ObjectName name = new ObjectName ("java.lang:type=GarbageCollector,name=Copy"); Object val = mconn.getAttribute(name,"CollectionCount"); System.out.println(name + "\n\tCollectionCount=" + val); } This example queries an MBean provided by the JVM. Each MBean has a multi-part name. This code converts the text form of the name into the internal form. DomainOne or more property/value pairs, separated by commas An object name consists of :

Java Performance Analysis 301 Page 44 Dec. 7, 2006 JMX Example Program public static void main(String[] args) throws Exception { JMXServiceURL url = new JMXServiceURL( "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi"); JMXConnector jconn = JMXConnectorFactory.connect(url); MBeanServerConnection mconn = jconn.getMBeanServerConnection(); ObjectName name = new ObjectName ("java.lang:type=GarbageCollector,name=Copy"); Object val = mconn.getAttribute(name,"CollectionCount"); System.out.println(name + "\n\tCollectionCount=" + val); } This example queries an MBean provided by the JVM. Get the value of an attribute and print it. Note that we never get the MBean object.

Java Performance Analysis 301 Page 45 Dec. 7, 2006 MBeanServerConnection Methods queryNames Gets names that match namespace & property criteria getMBeanInfo Find out what attributes, methods and events are supported get/setAttribute(s) Gets or sets one or more attribute values add/removeNotificationListener Register or unregister to be notified of certain events create/unregisterMBean Create or remove an Mbean invoke Call an operation on an Mbean

Java Performance Analysis 301 Page 46 Dec. 7, 2006 JBoss Application Server JBoss Application Server is based on a JMX kernel* JMX is used extensively * Up through 4.x, JBoss AS 5.0 uses an inversion-of-control microkernel Diagram is from The JBoss 4 Application Server Guide, Chapter 2

Java Performance Analysis 301 Page 47 Dec. 7, 2006 JMX-Console

Java Performance Analysis 301 Page 48 Dec. 7, 2006 JMX-Console Attributes Operations Changeable Invokeable

Java Performance Analysis 301 Page 49 Dec. 7, 2006 Twiddle Command line script for JBoss Application Server Used to interact with MBean Server Connection twiddle opt command arguments Arguments vary based on the command The action to perform. Ex: get, settattrs, invoke Command options Ex: -s

Java Performance Analysis 301 Page 50 Dec. 7, 2006 Twiddle Example twiddle –s somehostname get "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool" InUseConnectionCount NOTE: Twiddle command must be all on one command line.

Java Performance Analysis 301 Page 51 Dec. 7, 2006 Twiddle Example twiddle –s somehostname get "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool" InUseConnectionCount The host running the application server NOTE: Twiddle command must be all on one command line.

Java Performance Analysis 301 Page 52 Dec. 7, 2006 Twiddle Example twiddle –s somehostname get "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool" InUseConnectionCount Get the value of one attribute from one MBean NOTE: Twiddle command must be all on one command line.

Java Performance Analysis 301 Page 53 Dec. 7, 2006 Twiddle Example twiddle –s somehostname get "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool" InUseConnectionCount The name of the MBean. In this example, a database connection pool. NOTE: Twiddle command must be all on one command line.

Java Performance Analysis 301 Page 54 Dec. 7, 2006 Twiddle Example twiddle –s somehostname get "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool" InUseConnectionCount The name of the attribute NOTE: Twiddle command must be all on one command line.

Java Performance Analysis 301 Page 55 Dec. 7, 2006 Twiddle Example twiddle –s somehostname get "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool" InUseConnectionCount NOTE: Twiddle command must be all on one command line. Result: InUseConnectionCount=4

Java Performance Analysis 301 Page 56 Dec. 7, 2006 JMX Example Application for JBoss AS public static void main(String[] args) throws Exception { Hashtable env = new Hashtable(); String factory="org.jnp.interfaces.NamingContextFactory"; env.put(Context.INITIAL_CONTEXT_FACTORY, factory); String url1 = "jnp://localhost:1099"; env.put(Context.PROVIDER_URL, url1); Context ctx = new InitialContext(env); MBeanServerConnection mconn =(MBeanServerConnection) ctx.lookup("jmx/invoker/RMIAdaptor"); ObjectName name = new ObjectName( "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool"); Object val=mconn.getAttribute(name,"InUseConnectionCount"); System.out.println(name+"\n\tInUseConnectionCount="+val); }

Java Performance Analysis 301 Page 57 Dec. 7, 2006 JMX Example Application for JBoss AS public static void main(String[] args) throws Exception { Hashtable env = new Hashtable(); String factory="org.jnp.interfaces.NamingContextFactory"; env.put(Context.INITIAL_CONTEXT_FACTORY, factory); String url1 = "jnp://localhost:1099"; env.put(Context.PROVIDER_URL, url1); Context ctx = new InitialContext(env); MBeanServerConnection mconn =(MBeanServerConnection) ctx.lookup("jmx/invoker/RMIAdaptor"); ObjectName name = new ObjectName( "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool"); Object val=mconn.getAttribute(name,"InUseConnectionCount"); System.out.println(name+"\n\tInUseConnectionCount="+val); } The Java Naming and Directory Interface (JNDI) is used to look up the MBean Server Connection. This is boilerplate, only line that changes is the URL for the server (localhost).

Java Performance Analysis 301 Page 58 Dec. 7, 2006 JMX Example Application for JBoss AS public static void main(String[] args) throws Exception { Hashtable env = new Hashtable(); String factory="org.jnp.interfaces.NamingContextFactory"; env.put(Context.INITIAL_CONTEXT_FACTORY, factory); String url1 = "jnp://localhost:1099"; env.put(Context.PROVIDER_URL, url1); Context ctx = new InitialContext(env); MBeanServerConnection mconn =(MBeanServerConnection) ctx.lookup("jmx/invoker/RMIAdaptor"); ObjectName name = new ObjectName( "jboss.jca:name=jdbc/TheDS,service=ManagedConnectionPool"); Object val=mconn.getAttribute(name,"InUseConnectionCount"); System.out.println(name+"\n\tInUseConnectionCount="+val); } Rest of the code is same as from before, except for the MBean name & attribute.

Java Performance Analysis 301 Page 59 Dec. 7, 2006 Interesting MBeans jboss.jca:service=ManagedConnectionPool,name= Number of database connections (available, in use, max, min) jboss.management.local:J2EEServer=local,j2eeType=,J2EEApplication=,EJBModule=,name= -or- jboss.management.local:J2EEServer=local,j2eeType=servlet,J2E EApplication=,WebModule=,name= Stats – method hit counts and timings (min, max, average) Important: see the white paper on using Stats with twiddle! jboss.web:type=ThreadPool,name=http- - Thread counts, aka client connections (min, max, spare, current) And many, many more...

Java Performance Analysis 301 Page 60 Dec. 7, 2006 Conclusions You should now know how to Use JConsole to monitor a variety of JVM data Use JStat to monitor JVM heap and class statistics Use JMX (via key MBeans) to monitor Java applications, including application servers

10/20/06Java Performance Analysis 301 Page 61 Thank you! Questions?