1 Automating Monitoring with Puppet Chris Mague Moovweb May 23, 2012

2 Where I Want to be

3 What I'll Settle For.....

4 Requirements Rock solid stability Automated node addition (discovery) Scales horizontally Service dependency models Easy to write plugins Promotes sane workflows Unified front end view Flexible configuration

5 Tool Stack

6 What???? but #monitoringsucks and #ihatenagios How could you?

7 In defense of Nagios Been around since 1996 Has Service dependencies Easy to write plugins Easy-ish to troubleshoot ROCK SOLID

8 Valid attacks on Nagios No automated discovery It's complicated to setup Text files – really? Front end won't win any beauty contests Development is slow Stats collection is a PITA

9 Solutions Use Icinga! Use Puppet to auto configure Stats – leave it to graphite. It's really good at that Big boys and girls learn their tools

10 Icinga Fork of Nagios Configurations are compatible More solid architecture ( core, API, Web, IDODB ) Nice front end, nice mobile front end Can use NRPE

11 High Level View

12 Configure Icinga Servers using Puppet Standard Types

13 Things to configure with Standard Types icinga.cfg (file) => icinga main config file Apache icinga.conf (file) => http access to each server cgiauth.cfg (file) => cgi access cgi.cfg (file) => options, users templates.cfg (file) got lazy => use for basic classes idomod.cfg (template) => template for hostname to DB

14 Configure Icinga using Nagios Types

15 Puppet Nagios Types nagios_command nagios_contact nagios_contactgroup nagios_host nagios_hostdependency nagios_hostescalation nagios_hostextinfo nagios_hostgroup nagios_service nagios_servicedependency nagios_serviceescalation nagios_serviceextinfo nagios_servicegroup nagios_timeperiod

16 Configuring Hosts

17 Overview

18 Detailed Overview

19 Store Configs Store puppet info in a DB Retrieve information from DB Share info across nodes Use thin_storeconfigs Set up on puppet master

20 Exporting Nagios_host Resources Export = Save to DB Use facter for dynamic data PRO TIP: use ENC PRO TIP: use targets PRO TIP: hostgroups PRO TIP: use tags

21 PRO TIP: Use your ENC

22 PRO TIP: use targets Use cfg_dir in icinga.cfg Create a unique file per host or service Addition and removal are now super easy Also default dirs are in a horrible place /etc/nagios

23 PRO TIP: hostgroups Add machines to a hostgroup Add services to a hostgroup New machines inherit all of the services associated with a hostgroup

24 PRO TIP: use tags Tags allow you to filter resources so that you only realize those resources that you need

25 Configuring Services/Commands

26 Icinga Services OR 'Stuff I want to monitor' Associate with a hostgroup Use a target

27 Icinga Commands OR 'What actually gets run' Use Macros to set paths in resource.cfg

28 Dependencies

29 PRO TIP: Dependencies Unreliable services Cut down on the number of alerts Tell me what's really wrong Route alerts accordingly

30 Nagios_servicedependencies

31 NRPE

32 NRPE Runs on client Secured via SSL Has ACLs Runs as nobody Can run commands Useful for other things...

33 Configuring NRPE

34 NRPE Checks

35 Plugins

36 exchange.nagios.org

37 Writing Plugins Write in any language Output 1 line to stdout NRPE/Icinga/Nagios all use exit codes to determine status Run by hand to check

38 Workflows

39 Watching Monitoring

40 Scheduling Downtime

41 Filtering

42 Alerting

43 #monitoringisawesome REMOVE unreliable checks Just MONITOR – don't bolt on - especially stats TIER your monitoring Use timeperiods for sanity Delegate responses Use dependencies to pin down problems quickly Work smart

44

45 Resources Icinga Puppet NRPE IRC ##infra-talk, #icinga, #puppet #gaijin (freenode), Thanks Yvonne Kong, Michael Catlin, Juan Ortega, Anthony Kong, Puppet Labs, Icinga Team