Developing 14,000+ messages per second IoT systems No Rocket Scientist required! Developing 14,000+ messages per second IoT systems

Developers Audience Building High Through-put Systems on CF Wanting To Monitor Apps on CF Interested In Significant Load to test Apps Needing a Recovery Area Arity Sensor and Insights eXchange project – A_SIX Developers building high throughput systems using Cloud Foundry Developers wanting to monitor their applications running in Cloud Foundry Developers interested in generating significant load to test applications Developers needing a recovery area after this mornings Pancake Stack breakfast

Who Am I? Dale Robinson A_Six Anchor Arity Employed by Arity Arity provides API services in the telematics sensor market for companies such as Allstate, Answer Financial, Esurance. Anchor for an XP team, working on the Arity Sensor and Insights eXchange project – A_SIX

A_SIX application performance But we wanted… A_TEN Used https://www.lizenzhinweisgenerator.de/?lang=en to create the attribution for the Chris Hemsworth image (as per https://commons.wikimedia.org/wiki/Commons:Reusing_content_outside_Wikimedia#Reuse_assistance_tools ). Image is found at: https://commons.wikimedia.org/wiki/File:Chris_Hemsworth_Thor_2_cropped.png

JMeter from Java Left – Spring Boot (v1.5.3) JMeter project files. Highlighted file is the JMeter configuration file Right top – contents of jmeter.yml Right centre – JmeterPropertyLoader.java file Right bottom – build.gradle dependency section

JMeter from Java @RestController class – JmxProcessor.java

Monitoring Apps in Cloud Foundry VisualVM Enable JMX in (Pivotal) Cloud Foundry cf set-env <appName> JBP_CONFIG_JMX '{enabled: true}’ Use cf ssh to connect to the container cf ssh -N -T -L 5000:localhost:5000 <appName> Create a JMX connection within VisualVM

Monitoring - VisualVM

Monitoring - VisualVM

Monitoring - VisualVM

Monitoring VisualVM – What’s missing? Workarounds: Heap dumps MBeans Spring Actuators

Monitoring - MBeans

Monitoring - MBeans

Monitoring - Spring Spring Actuators Confirm they are enabled/sensitive /dump to produce JSON of thread dump /heapdump to produce GZIP hprof file /metrics to produce JSON of container data /env to produce JSON of JVM details See Spring documentation for more…

Monitoring conclusion Heap dump considerations Spring Actuator vs. MBeans Heap dumps are saved to the container Spring approach will automatically download MBean approach requires manual SCP Specify additional disc space accordingly This is true for both the MBean and Spring options

Micro-services are good… A_SIX A_SIX API API Persistence Persistence Insights Insights Subscriptions Subscriptions Republisher Republisher

Asynchronous processing and IO Thread pool is unconstrained Decoupling reading and writing Error handling

Spring Boot configuration Tomcat embedded server defaults 200 Threads 10,000 Connections OS Connection Queue 100 Keep Alive Requests

Spring Boot configuration

Spring AMQP configuration

Spring Boot configuration So now you can, but should you? Probably not…. The defaults are very good Only if you are certain of the issue Prefer the sledgehammer of more instances

Security Driven Development You load tested using JMeter You profiled using VisualVM You employed micro-services You tweaked Spring Boot / Tomcat Your application is flying... …and you just killed Cloud Foundry

Bits and bobs… cf curl /v2/apps/$(cf app <app> --guid)/stats Swapping Tomcat for Jetty or Undertow AsyncRabbitTemplate and AsyncRestTemplate PCFDev

Links https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html https://github.com/desRobinson https://medium.com/netflix-techblog/tuning-tomcat-for-a-high-throughput-fail-fast-system-e4d7b2fc163f https://blog.jayway.com/2014/09/09/asynchronous-spring-service/ http://zoltanaltfatter.com/2016/09/26/async-rabbit-template-with-spring-boot/ https://pivotal.io/pcf-dev http://sykesm.mybluemix.net/posts/jmx-in-diego/

Thanks!