Developing annotation based monitoring framework Fedor Romanov, TomTom

Monitoring distributed back-end a request takes 527ms to complete—where to look for clues? © 2015 TomTom. All rights reserved. Confidential information. 2

Monitoring avoids troubleshooting inspect individual transactions 1. An incoming web request is dispatched to the component responsible for generating the response. 2. The request may need to be authenticated against an LDAP server. 3. The controller component makes queries against a database. 4. The controller component also makes calls to a third-party web service. 5. The controller component aggregates all of the retrieved data into a set of business objects for display. 6. The business objects are rendered and the response is sent to the user's browser. 7. AJAX code running in the browser causes additional requests to be sent back to the server. © 2015 TomTom. All rights reserved. Confidential information. 3

Collecting the stats 4 What types of statistics can be gathered Application Response Time time per request Database Response Time time per request Errors errors within application Garbage Collection time spent in GC Thread State active thread states Memory Allocation heap and PermGen © 2013 TomTom. All rights reserved. Confidential information. Method ServiceA::doA() was called 302 times and consumed on avg 100ms Method ServiceB::doB() was called 53 times and consumed on avg 1200ms

Use JDK tools at hand 5 Too generic and low-level 1.Logs 2.JVisualVM 3.JStack 4.JMap 5.App server Monitoring 6.JConsole © 2013 TomTom. All rights reserved. Confidential information.

Modern APM tools 6 make monitoring easier Targeted a different type of developers from enterprise to startups Both use Java agent to collect performance data. Not the best fit for microservices, need to see instantly what is taking so long. © 2013 TomTom. All rights reserved. Confidential information.

Requirements for TT monitoring fw 7 Avoid high coupling with 3d partly libs Easy to set-up as use. No DevOps needed. Low performance impact Non-intrusive: Easy monitoring of entire methods by simply adding an annotation Ability to disable and reset accumulated data Easily swap underlying monitoring implementation(should be open source) Spring support Standard deviation © 2013 TomTom. All rights reserved. Confidential information.

Why non-intrusiveness matters 8 Errors during monitoring cannot affect an application Adding performance logging statements in code reduces “signal-to-noise” ratio. Not safe, can throw NullPointerException: Stopwatch stopwatch = SimonManager.getStopwatch("sampled- stopwatch"); System.out.println("Stopwatch sample: " + stopwatch.sample()); © 2013 TomTom. All rights reserved. Confidential information.

Exploring available tools 9 Establishing common functionality 1.Javasimon simon simon 1.JAMon Metrics rics rics © 2013 TomTom. All rights reserved. Confidential information. Based on intersection, the interfaces have been defined and implemented.

Base notions 10 3 paramount interfaces under the hood Counter, for counting events. Has methods: void increase() void descrease() int getValue() Stopwatch, for timing events. Has methods: Split start() void addTime(long) Split, a subcomponent of the Stopwatch, representing a running Stopwatch. Contains: long stop() long runningFor() © 2013 TomTom. All rights reserved. Confidential information.

Client annotations 11 How it looks from actual code AnnotationApplied toUsageHow does it count method calls. uses AOP to run code before the a Counter or Stopwatch monitor into a field. uses Spring to find fields using the annotation, injects a corresponding object into the time a monitor. uses AOP to run code before and after the method. © 2013 TomTom. All rights reserved. Confidential information.

Time to get to the code 12 © 2013 TomTom. All rights reserved. Confidential information.

Thank you Any questions?