1. Monitoring IBMi by Nagios Core & Nagiosgraph
Václav Matoušek
Consultant, IBM CZ

2. Agenda

3. Nagios Overview What Is Nagios? What Nagios Provides
Nagios is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes.
What Nagios Provides
Designed with scalability and flexibility in mind, Nagios gives you the peace of mind that comes from knowing your organization’s business processes won’t be affected by unknown outages.
Nagios is a powerful tool that provides you with instant awareness of your organization’s mission-critical IT infrastructure. Nagios allows you to detect and repair problems and mitigate future issues before they affect end-users and customers.
By using Nagios, you can:
Plan for infrastructure upgrades before outdated systems cause failures
Respond to issues at the first sign of a problem
Automatically fix problems when they are detected
Coordinate technical team responses
Ensure your organization’s SLAs are being met
Ensure IT infrastructure outages have a minimal effect on your organization’s bottom line
Monitor your entire infrastructure and business processes

4. Nagios Overview How It Works
Monitoring
IT staff configure Nagios to monitor critical IT infrastructure components, including system metrics, network protocols, applications, services, servers, and network infrastructure.
Response
IT staff can acknowledge alerts and begin resolving outages and investigating security alerts immediately. Alerts can be escalated to different groups if alerts are not acknowledged in a timely manner.
Maintenance
Scheduled downtime prevents alerts during scheduled maintenance and upgrade windows.

5. Nagios Overview How It Works continuing
Alerting
Nagios sends alerts when critical infrastructure components fail and recover, providing administrators with notice of important events. Alerts can be delivered via , SMS, or custom script.
Reporting
Reports provide a historical record of outages, events, notifications, and alert response for later review. Availability reports help ensure your SLAs are being met.
Planning
Trending and capacity planning graphs and reports allow you to identify necessary infrastructure upgrades before failures occur.

6. Operating principle of Nagios

7. Nagios Plugins What Is A Plugin?
Plugins are standalone extensions to Nagios Core that make it possible to monitor anything and everything with Core.
Plugins process command-line arguments, perform a specific check, and then return the results to Nagios Core. They can be compiled binaries (written in C, C++, etc) or executable scripts (shell, Perl, PHP, etc).
In addition to the official Nagios plugins, you can find thousands of other plugins for monitoring all types of hardware, services, metrics, and applications at Nagios Exchange.

8. Favorite Nagios Projects
NRPE
NRPE allows you to remotely execute Nagios plugins on other Linux/Unix machines. This allows you to monitor remote machine metrics (disk usage, CPU load, etc.). NRPE can also communicate with Windows agent addons like NSClient++, so you can check metrics on remote Windows machines as well.
Nagiosgraph
Nagiosgraph is graphing tool that parses output and performance data from Nagios plugins and stores the data in RRD files. nagiosgraph displays data in Nagios trends, as popups for hosts and services, or in separate reports. Easy to set up and eminently customizable.
All favorite Nagios projects you can find on URL:

9. Monitoring IBMi using Nagios Design
check_nrpe
NRPE
CPU Utilization
Active Users
Total Jobs
Local Resources and Services
Monitoring Host
Remote IBMi Host
The NRPE addon consists of two pieces:
The check_nrpe plugin, which resides on the local monitoring machine
The NRPE daemon, which runs on the remote IBMi machine
When Nagios needs to monitor a resource or service from a remote IBMi machine
Nagios will execute the check_nrpe plugin and tell it what service needs to be checked
The check_nrpe plugin contacts the NRPE daemon on the remote host over an (optionally) SSL protected connection
The NRPE daemon runs the appropriate Nagios plugin to check the service or resource
The results from the service check are passed from the NRPE daemon back to the check_nrpe plugin, which then returns the check results to the Nagios process.

10. Nagios demonstration

11. Monitoring Host Server or partition with Linux (x86 or ppc)
Installation of Nagios Core or Nagiosgraph described in many articles like
SAVF with NWSSTG of Linux partition with installed Nagios Core and Nagiosgraph will be available together with this presentation

12. NRPE installation on IBMi Install required LPP‘s
Install 5733OPS IBMi Open Source Solution
Install 5799PTL IBM Tools for Developer for i5/OS

13. NRPE installation on IBMi Download prerequisites
Attach: download-2.0.tar.zip (required)
Attach :setup2.sh — setup replace in download
Attach: wwwperzl.sh — wwwperzl.sh replace in download
from URL:
to directory /QOpenSys.

14. NRPE installation on IBMi Install prerequisites
Setup rpm and perzl *.deps (local copy)
call qp2term
> cd /QOpenSys
> jar xvf download-2.0.tar.zip
> tar -xf download-2.0.tar
> cp setup2.sh ./download
> cp wwwperzl.sh ./download
> cd /QOpenSys/download
> ./setup2.sh
(a) setup installs rpm.rte
(b) setup rpm install wget
(c) setup untar perzl deplists (
Note: (c) can take significant time (setting up prezl depend tree ...)
> ./wwwperzl.sh aix61 wget Nagios-nrpe
> ./wwwperzl.sh aix61 rpm Nagios-nrpe

15. NRPE installation on IBMi Create NAGIOS user
Use following command to create group profile NAGIOSGRP:
CRTUSRPRF USRPRF(NAGIOSGRP) PASSWORD(*NONE) TEXT(‘Group profile for NAGIOS’)
Use following command to create user NAGIOS:
CRTUSRPRF USRPRF(NAGIOS) PASSWORD(<password>) GRPPRF(NAGIOSGRP) TEXT(‘User profile for NAGIOS’)

16. NRPE installation on IBMi Download Pre-Compiled Binaries
You will need to download two packages of pre-compiled binaries to your IBM i server.
First download the pre-compiled Nagios plugin binaries for AIX 5.3 from the following URL:
Next download the pre-compiled NRPE binaries from the following URL:

17. NRPE installation on IBMi Create Directories
Login to your IBM i server as the privileged user. Go to PASE environment (call qp2term) and run the following commands: cd /usr/local mkdir nagios

18. NRPE installation on IBMi Unpack Binaries
Next unpack the pre-compiled plugins and place the extracted files into the /usr/local/nagios directory using commands similar to the following:
cd /tmp
gzip –d /path/to/aix53_nrpe-nsca-plugins.tgz
tar xvf /path/to/aix53_nrpe-nsca-plugins.tar
cd /tmp/nagios
cp -R * /usr/local/nagios
Next, unpack the precompiled NRPE binaries using commands similar to the following:
gzip –d /path/to/AIX-5.3-nrpe-2.12-binaries.tar.gz
tar xzf /path/to/AIX-5.3-nrpe-2.12-binaries.tar
cp usr/local/nagios/bin/nrpe /usr/local/nagios/bin
cp usr/local/nagios/etc/nrpe.cfg /usr/local/nagios/etc

19. NRPE installation on IBMi Edit nrpe.cfg file
Change location of nrpe.pid file to:
# PID FILE
# The name of the file in which the NRPE daemon should write it's process ID
# number. The file is only written if the NRPE daemon is started by the root
# user and is running in standalone mode.
pid_file=/usr/local/nagios/var/run/nrpe.pid
Add IP address of Nagios server to allowed hosts using:
# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon. #
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
allowed_hosts= ,

20. NRPE installation on IBMi Set Permissions
Create empty nrpe.pid file: touch /usr/local/nagios/var/run/nrpe.pid Change folder permissions so NRPE will operate properly, with the following command: chown -R nagios /usr/local/nagios

21. NRPE installation on IBMi Specify NRPE Port Number
Next, add nrpe service with specific port number for NRPE: ADDSRVTBLE SERVICE(nrpe) PORT(5666) PROTOCOL(tcp) TEXT(‘#NRPE’)

22. NRPE installation on IBMi Create NRPE.PGM
Next, create program for starting nrpe deamon:
Signon as nagios user.
CRTLIB LIB(NAGIOS)
CRTSRCPF FILE(NAGIOS/QCLSRC) RCDLEN(112) MBR(NRPE) TEXT(‘Start NRPE Deamon’)
Use following command to modify NRPE member:
WRKMBRPDM FILE(NAGIOS/QCLSRC)
Change type of NRPE member to CLP. Then use Option 2 to add following as a content of member:
PGM
CALL PGM(QP2SHELL) +
PARM('/usr/local/nagios/bin/nrpe' '-c‘ '/usr/local/nagios/etc/nrpe.cfg' +
'-n' '-i')
ENDPGM
Before add this text, use F13 to change Uppercase Input Only to N.
Use F3 to Save then Enter to update member. Then use Option 14 to Compile member and then Option DM to see message about result of Compiling.

23. NRPE installation on IBMi Start NRPE Deamon
Open *INETD configuration file to add NRPE:
EDTF(‘/QIBM/ProdData/OS400/INETD/inetd.conf’)
Add following text to the end of file:
#NRPE
nrpe stream tcp nowait NAGIOS /QSYS.LIB/NAGIOS.LIB/NRPE.PGM #
Use F2 to Save and F3 to Exit.
Restart *INETD TCP Server using following commands:
ENDTCPSVR *INETD
STRTCPSVR *INETD

24. Plugins for IBMi Create Plugin
Plugin for checking number of active jobs on IBM – example: created on base DB2 for i Services SELECT TOTAL_JOBS_IN_SYSTEM, MAXIMUM_JOBS_IN_SYSTEM, ACTIVE_JOBS_IN_SYSTEM, INTERACTIVE_JOBS_IN_SYSTEM, ELAPSED_TIME FROM TABLE(QSYS2.SYSTEM_STATUS(RESET_STATISTICS=>'YES')) X"

25. Plugins for IBMi Example code in Python
/usr/local/nagios/libexec/check_ibmi_jobs.py #!/usr/bin/python -tt import ibm_db conn = ibm_db.connect("*LOCAL", "<user>", "<password>") sql = "SELECT TOTAL_JOBS_IN_SYSTEM, MAXIMUM_JOBS_IN_SYSTEM, ACTIVE_JOBS_IN_SYSTEM, INTERACTIVE_JOBS_IN_SYSTEM, ELAPSED_TIME FROM TABLE(QSYS2.SYSTEM_STATUS(RESET_STATISTICS=>'YES')) X" stmt = ibm_db.exec_immediate(conn, sql) result = ibm_db.fetch_both(stmt) while result != False: print(result["ACTIVE_JOBS_IN_SYSTEM"]) exit()

26. Plugins for IBMi Example code in bash
/usr/local/nagios/libexec/check_ibmi_jobs.sh #! /bin/sh rv=$(/QOpenSys/usr/bin/python3 /usr/local/nagios/libexec/check_ibmi_jobs.py 2>&1) if [ $rv -lt 300 ] ; then status=0 statustxt=OK elif [ $rv -lt 500 ] ; then status=1 statustxt=WARNING else status=2 statustxt=CRITICAL fi echo "JOBS $statustxt: $rv active jobs|Active jobs=$rv;300;500;„ exit $status

27. Plugins for IBMi Add service to Nagios
Add following line to /usr/local/nagios/etc/nrpe.cfg command[check_ibmi_jobs]=/usr/local/nagios/libexec/check_ibmi_jobs.sh -w 300 -c 500 Restart nrpe deamon. Open session to Nagios server. Add following to /usr/local/nagios/etc/objects/services.cfg define service{ use generic-service,nagiosgraph host_name i5os3 service_description Total Jobs check_command check_nrpei!check_ibmi_jobs } service nagios restart

28. Plugins for IBMi Generic service example
define service{ name generic-service ; The 'name' of this service template active_checks_enabled 1 ; Active service checks are enabled passive_checks_enabled 1 ; Passive service checks are enabled/accepted parallelize_check 1 ; Active service checks should be parallelized (disabling this can lead to major performance problems) obsess_over_service 1 ; We should obsess over this service (if necessary) check_freshness 0 ; Default is to NOT check service 'freshness' notifications_enabled 1 ; Service notifications are enabled event_handler_enabled 1 ; Service event handler is enabled flap_detection_enabled 1 ; Flap detection is enabled process_perf_data 1 ; Process performance data retain_status_information 1 ; Retain status information across program restarts retain_nonstatus_information 1 ; Retain non-status information across program restarts is_volatile 0 ; The service is not volatile check_period 24x7 ; The service can be checked at any time of the day max_check_attempts 3 ; Re-check the service up to 3 times in order to determine its final (hard) state normal_check_interval 10 ; Check the service every 10 minutes under normal conditions retry_check_interval 2 ; Re-check the service every two minutes until a hard state can be determined contact_groups admins ; Notifications get sent out to everyone in the 'admins' group notification_options w,u,c,r ; Send notifications about warning, unknown, critical, and recovery events notification_interval 60 ; Re-notify about service problems every hour notification_period 24x7 ; Notifications can be sent out at any time register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE! } define service { name nagiosgraph action_url /nagiosgraph/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$ register 0

29. Create contact for alert sending
define contact{
contact_name nagiosadmin ; Short name of user
alias Nagios Admin ; Full name of user
service_notification_period x7
host_notification_period x7
service_notification_options w,u,c,r
host_notification_options d,r
service_notification_commands notify-service-by-
host_notification_commands notify-host-by-
; <<***** CHANGE THIS TO YOUR ADDRESS ****** }
define contactgroup{
contactgroup_name admins
alias Nagios Administrators
members nagiosadmin

30. Plugins for IBMi Another service available for download
Active users
based on DB2 for i – Services: QSYS2.ACTIVE_JOB_INFO
Attempt signon
based on DB2 for i – Services: QSYS2.USER_INFO
CPU Utilization
based on DB2 for i – Services: QSYS2.SYSTEM_STATUS_INFO
System ASP
XSM trafic
based on DB2 for i – Services: QSYS2.NETSTAT_INFO

31. Open source project
There is plan to create special www pages for this project
It is open do download or add any plugin to this project or ask to create new specific plugin

32. Děkuji za pozornost