1. Exploring Java Heap Dumps
Ryan Cuprak

2. Heap Contains Everything and can be DUMPED to DISK
Java Heap Review
Java objects are stored in the heap
All objects are globally reachable in the heap
Heap is created when an application starts
Size of heap is configured using –Xmx and –Xmx
Garbage collection prunes the heap and removes objects no longer reachable
Stack memory - variable values are stored when their methods are invoked
Heap Contains Everything and can be DUMPED to DISK

3. Why Analyze Heaps?
IT reports Java EE/Spring server memory footprint has grown to 9 gigs
Server app logs contain OutOfMemoryExceptions
Connections to queueing or database are exhausted
Serialized Java objects in queue are unreasonably large
Desktop application becomes unresponsive
Excessive amount of garbage collection

4. Java Heap Analysis
All you need is a profiler right?
Profiler
MAT

5. Memory Leaks
Textbook memory leaks - easy to find and fix.
JConsole

6. Production Heap Dumps 9 gigs of data 13k classes loaded
~ 136 million instances
~6,000 GC roots

7. Production Heap Dumps
Capture the Heap Dump and…

8. Production Heap Dumps

9. Heap Dump Panic Too much data! Impossible to comprehend
No human way to explore the data
Application data model is too complicated

10. Real Memory Leaks Bank Account: 1231209 Owner Bob Bank Account:
Owner
Julie
Bank Account:
Report
January 2018
Challenge:
Data looks good everywhere…
Owner
Bob
Report
January 2018

11. Real Memory Leaks Causes: Faulty clone methods Duplicate singletons
Accidently cached data
Cache logic bugs
Complications
May NOT GROW over time (leaks gets cleaned-up)
More than one non-trivial memory leak

12. What about OQL? OQL Downside
Object Query Language – used for querying heaps
SQL-like language
Supports JavaScript expressions
Supported in NetBeans and VisualVM
Downside
Poorly documented and hard to use
Easy to create runaway queries

13. Heap Analysis Solution
What?

14. NetBeans Profiler NetBeans is open source IDE/platform
Modular architecture
Clean code base
Profiler GUI
Profiler API

15. NetBeans Profiler API Parses hprof files
Creates an object model representing the hprof file
Pages data in from disk
Simple API (master in about 10 minutes)
Independent of NetBeans
Can be extract and use in any IDE – Plain old Java
Talk is really about how to build a custom heap analysis tool:
To answer specific data model questions
With custom logic for your data model

16. Generating Heap Dumps

17. Generating Heap Dumps Command line parameter: Command line: Ctrl-Break
-XX +HeapDumpOnOutOfMemoryError
Command line:
jmap –dump:format=b,file=dump.hprof <pid>
jhsdb jmap --binaryheap --pid <pid>
jcmd <pid> GC.heap_dump <file name>
Ctrl-Break
Command Line

18. Generating Heap Dumps
Programmatic

19. Generating Heap Dumps
APM tools
JMX

20. Heap Dump Warning Dumping the heap: Takes time Consumes diskspace
Negatively affects performance

21. Targeted Heap Dumps
Serialize object graphs from application to a file.
Read the serialized data into another tool and then programmatically create a heap dump.

22. Building a Profiler

23. Building Custom Profiler
Create NetBeans Platform App
Copy API src out of NetBeans

24. NetBeans Platform App

25. NetBeans Platform App
Add dependency on “Java Profiler (JFluid)”

26. Profiler Sources Checkout source: Profiler code:
Profiler code:
netbeans/profiler/lib.profiler/src/netbeans/lib.profiler/heap
Copy heap directory

27. Which Approach? Copying sources easiest
Most analysis apps are command line (one-offs)
- Note -
You don’t need the classpath of the application from which the heap was generated.

28. NetBeans Profiler API

29. Opening a Heap
That’s All!

30. Heap Object Methods
getJavaClassByName(String fqn) : JavaClass getAllClasses() : List getBiggestObjectsByRetainedSize(int number) : List getGCRoots(): GCRoot getInstanceByID(long instanceId) : Instance getJavaClassByID(long javaclassId) : JavaClass getJavaClassesByRegExp(String regexp) : HeapSummary getSummary() : Properties

31. System Properties

32. Heap Summary getTotalLiveInstances() : long getTime() : long
getTotalAllocatedBytes() : long
getTotalAllocatedInstances() : long
getTotalLiveBytes() : long

33. Exploration Starting Points
GCRoots
Threads (really GCRoots)
Class Types

34. GC Roots
Garbage Collection Root is an object that is accessible from outside the heap.
Objects that aren’t accessible from a GC Root are garbage collected
GC root categorization:
Class loaded by system class loader
Thread
Stack Local
Monitor
JNI Reference
Held by JVM

35. Garbage Collection Roots
Java frame: 44
thread object: 5
JNI global: 29
sticky class: 1284

36. GCRoot Objects

37. GC Roots root.getKind() : String
JNI_GLOBAL = "JNI global"; JNI_LOCAL = "JNI local"; JAVA_FRAME = "Java frame"; NATIVE_STACK = "native stack"; STICKY_CLASS = "sticky class"; THREAD_BLOCK = "thread block"; MONITOR_USED = "monitor used"; THREAD_OBJECT = "thread object"; UNKNOWN = "unknown"; INTERNED_STRING = "interned string"; FINALIZING = "finalizing"; DEBUGGER = "debugger"; REFERENCE_CLEANUP = "reference cleanup"; VM_INTERNAL = "VM internal"; JNI_MONITOR = "JNI monitor";
root.getKind() : String

38. Finding Classes Can perform lookup using: IDs are unique to heap dump
Fully qualified class name (ex. java.lang.String)
Class ID
Instance ID
IDs are unique to heap dump
Hash codes are not available!

39. Profiler Data Model JavaClass Instance A Instance B Value Value Value

40. Class
Java.lang.String
Java.util.List

41. Instances From an instance: Who references the instance
Who does the instance reference
Perform instanceof to find out:
ObjectArray
PrimitiveArray
GCRoot can take forever…

42. Values
If you ask an instance for its references, you get a list of Value objects.

43. Example: Member Variables
Iterates over all Person objects and prints member variables.

44. Example: Static Variables

45. Example: References

46. String Implementation

47. String Extraction
Strings are objects – array of characters

48. LinkedList Implementation

49. LinkedList Implementation

50. LinkedList Extract

51. ArrayList

52. ArrayList

53. Thread Extraction

54. Noise: Ignore Internal Classes
Ignore internal JVM classes

55. Puzzler Prints: Count: 231
230 entries have 1 are referenced by one other object
1 entry is “owned” by 822 other objects

56. Demo App Exploration

57. Demo App
Note: Used HashSets, Arrays[][], Lists, and Vectors

58. Demo App

59. Demo App

60. Demo App

61. String Utilization 5159 Strings in heap dump
15 associated with data model

62. String Utilization
Output

63. Data Model Leak
Add logic to fire an employee:

64. Data Model Leak
Fired Adam – shouldn’t be in the system!

65. Data Model Leak
Found!

66. Data Model Leak
Leaking here!

67. Best Practices Be mindful of your heap Heap processing is I/O bound
Cache analysis on disk when processing large heaps
Heap processing is I/O bound
Not all profiler calls are the same
Look for Javadoc: Speed: normal
Maintain a list of processed objects
Easy to run in circles
Exclude JVM internal classes from analysis
Revisit graph algorithms!

68. Summary Heap snapshot can be easily explored
Excellent way to verify application logic
Only way to identify deep data model/logic errors
Can be used to recover data
Generate a heap snapshot from a frozen/corrupted application and then mine

69. Q&A
/ Blog: cuprak.info Linkedin: Slides: