Memory issues in production systems

Production system Restricted access Application, DB, Application server, log files Debugging, monitoring Investigation should not affect users Reproducing memory issue is not the best idea

Memory leaks What is memory leak java.lang.OutOfMemory: heap size other symptoms - extensive garbage collection Hard to discover Users don't recognize it Hard to investigate Few information Sometimes hard to fix Design issues

Investigation of memory issues Log files application logs, access logs, gc logs Monitoring behavior of running application JConsole Heap dump Analysis JHAT, MAT, Netbeans

Log files Motivation find issues find patterns leading to issues Problems combining different logs (access log, application log) bad visibility of root cause – not visible or hidden in big amount of minor issues memory problems trigger only few types of messages

Investigating log files Application logs Log free memory once per minute Access logs Analysis of traffic and business use cases GC logs Show problems with GC (stop the world)

Monitoring via JMX Tools JConsole Motivation More information than in logs Monitoring tests Problems Not always applicable on production system (security reasons) Test might not cover all scenarios

Heap dump analysis Tools JMAP, JHAT, MAT, Netbeans Motivation Offline investigation of behavior during specific time frame Comparison of different heap dumps Problems Access to heap dumps (security)

Creating Heap dump jmap -dump:live,format=b,file=heap.dump Part of JDK Used to create heap dump on demand JVM parameters -XX:+HeapDumpOnOutOfMemoryError - XX:HeapDumpPath=/directory/subdirectory Produces file java_pid.hprof Used for production system, creates heap dump when OutOfMemory is thrown

Tools for heap dump analysis JHAT Part of JDK 6 Running as web application IDE Netbeans, MAT (Eclipse plugin) Commercial tools

Heap dump analysis - JHAT Heap Histogram List of classes with number of instances and used memory Referrers vs Referees Navigating in object graph Example: Form → String → [c

Heap dump analysis – Netbeans, MAT Dominator tree Object x dominates object y if every path in object graph from root to y goes through x Shallow vs Retained heap Shallow heap is memory used by object Retained set of X is set of objects which would be removed by GC if X is removed by GC Retained heap is sum of shallow heap in retained set Other queries Leak suspects Top consumers

Heap dump Analysis - OQL Object query Language Different dialect for each tool JHAT  Simple SQL combined with javascript  callback functions MAT  More SQL like (Subquery, union, distinct)  Predefined functions (dominator)

OQL Examples JHAT select heap.findClass('org.apache.catalina.util.ServerInfo').statics.serverNumber.toString() MAT SELECT serverNumber.toString() FROM OBJECTS org.apache.catalina.util.ServerInfo

Experiences Bugs Incorrect usage of external libraries or frameworks Design issues

Bugs

Usage of frameworks Hibernate Short time usage of big amount of memory Too big first level cache Good understanding of framework is a must

Design Issues Overused session scope Big pages Big buffer in page context Page context referenced by tag handlers Heavy load causes big memory usage Fix might require changing use cases Reusing functionality

Interesting links OQL help for JHAT Standalone MAT