THE ART AND METHODOLOGY OF TROUBLESHOOTING Alexey Diomin,

Production

Agenda  Logging  Offline analysis  Online analysis

Monitoring  System monitoring  Zabbix  Nagios  Application Monitoring  New Relic  Logstash  Kibana

Zabbix. CPU load example

Zabbix. Memory example

Zabbix. CMS Old Gen

Logstash. Kibana

“We need to go deeper”

 jstack (thread dump)  jstack  kill -QUIT

jstack  Very simple profiler  Lock detection

jstack example

“We need to go deeper”  jstack (thread dump)  jstack  kill -QUIT  jmap  jmap -dump  jmap -histo

jmap  Eclipse Memory Analyzer (MAT)   jvisualvm 

MAT

Leak Detection

RingBuffer

MAT

“We need to go deeper”  jstack (thread dump)  jstack  kill -QUIT  jmap  jmap -dump  jmap -histo  JFR / JMC

JFR

Event Type

JFR Basic features:  Code Profile  Allocation profile

JFR Basic features:  Code Profile  Allocation profile Killer features:  Contentions  Exceptions rate

Code profile

Allocation profile

Netty’s bug example

Contention

Hbase latency

Contentions example

Exceptions rate

Context Switch

Exception example  com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException 1. “You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ….” SQL syntax for prepared statements does not support multi-statements (that is, multiple statements within a single string separated by “;” characters). 2. “Can't create more than max_prepared_stmt_count statements” useServerPrepStmts Use server-side prepared statements if the server supports them

useServerPrepStmts

Summary  Logging  Offline analysis  Online analysis

Links  Java Performance: The Definitive Guide   Troubleshooting Guide for Java SE 6 with HotSpot VM  pdf pdf

Black boxes? Not in troubleshooting!!!