1. THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin, diominay@gmail.com

2. Production

3. Agenda  Logging  Offline analysis  Online analysis

4. Monitoring  System monitoring  Zabbix  Nagios  Application Monitoring  New Relic  Logstash  Kibana

5. Zabbix. CPU load example

6. Zabbix. Memory example

7. Zabbix. CMS Old Gen

8. Logstash. Kibana

9. “We need to go deeper”

10.  jstack (thread dump)  jstack  kill -QUIT

11. jstack  Very simple profiler  Lock detection

12. jstack example

13. “We need to go deeper”  jstack (thread dump)  jstack  kill -QUIT  jmap  jmap -dump  jmap -histo

14. jmap  Eclipse Memory Analyzer (MAT)  http://www.eclipse.org/mat/ http://www.eclipse.org/mat/  jvisualvm  http://visualvm.java.net/ http://visualvm.java.net/

15. MAT

17. Leak Detection

18. RingBuffer

19. MAT

20. “We need to go deeper”  jstack (thread dump)  jstack  kill -QUIT  jmap  jmap -dump  jmap -histo  JFR / JMC

21. JFR

23. Event Type

24. JFR Basic features:  Code Profile  Allocation profile

25. JFR Basic features:  Code Profile  Allocation profile Killer features:  Contentions  Exceptions rate

26. Code profile

27. Allocation profile

28. Netty’s bug example

29. Contention

30. Hbase latency

31. Contentions example

32. Exceptions rate

33. Context Switch

34. Exception example  com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException 1. “You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ….” SQL syntax for prepared statements does not support multi-statements (that is, multiple statements within a single string separated by “;” characters). 2. “Can't create more than max_prepared_stmt_count statements” useServerPrepStmts Use server-side prepared statements if the server supports them

35. useServerPrepStmts

36. Summary  Logging  Offline analysis  Online analysis

37. Links  Java Performance: The Definitive Guide  http://shop.oreilly.com/product/0636920028499.do http://shop.oreilly.com/product/0636920028499.do  Troubleshooting Guide for Java SE 6 with HotSpot VM  http://www.oracle.com/technetwork/java/javase/tsg-vm- 149989.pdf http://www.oracle.com/technetwork/java/javase/tsg-vm- 149989.pdf

38. Black boxes? Not in troubleshooting!!!