1. Nagios Performance Tuning Nick Scott nscott@nagios.com

2. 20112 Abstract Topics To Be Covered Baseline System / Testing Method Implementation and Impacts of: RAM Disks Offloading MySQL with NDOUtils Memcached Systems Passive Checks Final Q & A

3. 2011 Nagios World Conference 3 Baseline System Used and Method of Analysis

4. 20114 Specs * - Unless otherwise noted

5. 20115 Method of Data Gathering iostat Harvesting Method iostat -x 60 60 > file.log Repeat twice After test was performed Reset virtual machine to clean state Chose log file with least extrema Allowed for two hours of normal operation Log files parsed and plotted

6. 20116 Description of Metrics Metrics Used For Comparison w/s r/s await avgqu-sz avgrq-sz

7. 2011 Nagios World Conference 7 RAM Disks

8. 20118 RAM Disk - Description Folder mounted in RAM Pros Very fast I/O Can act as separate volume to alleviate I/O issues on a particular volumes Cons Does not hold state on restart Uses actual RAM, not ideal for large files Miscalculation will cause more harm than good

9. 20119 RAM Disk - Applications Storing often accessed, small files objects.cache status.dat service-perfdata Beware of scaling issues Size can often become very large Computer treats it like any other RAM mounted volume, for better or for worse

10. 201110 RAM Disk - Implementation Implementation is very easy Near zero downtime required Calculate size Mount tmpfs / edit fstab Edit nagios.cfg Restart nagios service Does not increase points of failure

11. 201111 RAM Disk – Request Reads / Sec R = ~.852 of Vanilla Read Requests Per Second for Device sda Time

12. 201112 RAM Disk – Request Writes / Sec R = ~.98 of Vanilla Write Requests Per Second for Device sda Time

13. 201113 RAM Disk – Device Queue R = ~.556 of Vanilla Number of Items In Write Queue for Device sda Time

14. 201114 RAM Disk – Wait Queue R = ~.869 of Vanilla Average Wait (ms) Time

15. 201115 RAM Disk - Conclusion Improvement Will help overtaxed hard drives

16. 2011 Nagios World Conference 16 Offloading MySQL with NDOUtils

17. 201117 Offloading MySQL - Description Moves writing of database to separate server Server moved to does not have to be powerful Pros Alleviates I/O and CPU time that MySQL uses Creates a path for redundancy/high availability Cons Lights out Increased wait time on results Bandwidth utilization Requires additional hardware/virtualization

18. 201118 Offloading MySQL - Applications Alleviating CPU, I/O on central Nagios MySQL typically eats ~50% of CPU time MySQL and performance graphing on one machine causes incredible Disk I/O with larger installations Allows for scalability Clusterable Abstraction

19. 201119 Offloading MySQL - Implementation Implementation is invasive Requires downtime Requires additional server, additional software Adds possible point of failure Sends traffic over network Editing the config files ndo2db.cfg config.inc.php config.php

20. 201120 Offloading MySQL – Request Reads / Sec R = ~.76 of Vanilla Read Requests Per Second for Device sda Time

21. 201121 Offloading MySQL – Request Writes / Sec R = ~2.05 of Vanilla Write Requests Per Second for Device sda Time

22. 201122 Offloading MySQL – Average Size In Queue R = ~.707 of Vanilla Write Requests Per Second for Device sda Time

23. 201123 Offloading MySQL – Device Queue R = ~.55 of Vanilla Number of Items In Write Queue for Device sda Time

24. 201124 Offloading MySQL – Wait Queue R = ~.529 of Vanilla Average Wait (ms) Time

25. 201125 Offloading MySQL - Conclusion Definite improvement Nearly 50% reduction in hardware demand Smaller files in queue, but more writes

26. 2011 Nagios World Conference 26 Memcache

27. 201127 Memcache - Description Creates a collective of caches Can be one, or many memcache servers Avoids database hits Checks cache Hits database Updates database Removes from cache on update

28. 201128 Memcache - Description Pros Can run on most hardware Hardware can be most anything Allows for massive scalability Cons Can increase latency of requests Improper coding can cause incorrect returns Creates lots of network traffic

29. 201129 Memcache - Implemenation Installing on server hardware Generally in repositories Noninvasive, specialized Currently supported with Nagios XI

30. 201130 Memcache – Request Reads / Sec R = ~.15 of Vanilla Read Requests Per Second for Device sda Time

31. 201131 Memcache – Request Writes / Sec R = ~11.47 of Vanilla Write Requests Per Second for Device sda Time

32. 201132 Memcache – Average Size In Queue R = ~.27 of Vanilla Write Requests Per Second for Device sda Time

33. 201133 Memcache – Device Queue R = ~1.55 of Vanilla Number of Items In Write Queue for Device sda Time

34. 201134 Memcache – Wait Queue R = ~.259 of Vanilla Average Wait (ms) Time

35. 201135 Memcache - Conclusion Some tradeoffs Large reduction in reads Massive increase in writes Test did not accurately portray scalability

36. 2011 Nagios World Conference 36 Passive Checks

37. 201137 Passive Checks - Discussion Creates separation of labor Nagios Execution vs Remote Execution Bring some complexity Eases some cross-platform issues Allows for ridiculous checks Consumes more CPU cycles net Freshness checks are necessary, but counter- productive Forcing active checks anyways

38. 201138 Passive Checks - Discussion Common types and relative expense NSCA NRDP SNMP Ideal design

39. 201139 Final Thoughts Is the service very dynamic? Slow, semi-predictable, low priority Make it all from a little Have a way to profile your install