1. How to Monitor Ingres with Open Source Tools
Mark Whalley, Ingres Europe Ltd.

2. For information contact Product Management at products@ingres.com
OpenROAD XML
For information contact Product Management at
This presentation contains forward-looking statements that are based on management’s expectations, estimates, projections and assumptions. Forward-looking statements are made pursuant to the safe harbor provisions of the Private Securities Litigation Reform Act of 1995, as amended. These statements are not guarantees of future performance or functionality and involve certain risks and uncertainties, which are difficult to predict. Therefore, actual future functionality, features, results and trends may differ materially from what is forecast in forward-looking statements due to a variety of factors.
© 2010 Ingres Corporation
Slide 2
Slide 2
© Ingres 2010 2

3. Agenda Monitoring Ingres Why Monitor What to Monitor When to Monitor
Tools for Monitoring
Ingres Support Appliance / Nagios
Architecture Overview
Plug-ins
Guide to Building a Custom Plug-in
Example Plug-in
Questions and Answers
© 2010 Ingres Corporation

4. Monitoring Ingres
© 2010 Ingres Corporation

5. Why monitor Pre-empt problems
To keep installations up and running 24x7
Planned downtime
Trend analysis
Root cause analysis
The business expects it
Basis for performance tuning
Evidence for Service Level Agreements (SLA)
Auditing
Resource planning
Hardware growth
Archiving
© 2010 Ingres Corporation

6. What to Monitor Ingres Processes Logs Databases Connectivity Locking
Space utilisation
Applications
User access
Resource utilisation
Process state
© 2010 Ingres Corporation

7. What to Monitor Hardware CPU load Swap space Disk fragmentation
Network
Inaccessible server
Performance
© 2010 Ingres Corporation

8. When to Monitor 24 hours a day 7 days a week 52 weeks of the year
It is impossible to manually monitor large scale enterprises, but the need is to monitor them all of the time.
© 2010 Ingres Corporation

9. Closed Source Tools for Monitoring
CA Unicentre
HP OpenView
BMC Patrol
Others
© 2010 Ingres Corporation

10. Open Source Tools for Monitoring
Ingres utilities
ipm, logstat, lockstat, ima, log files, snmp, ...
Operating system utilities
ps, top, sar, df, ...
Ingres Support Appliance (ISA) / Nagios
Others
Many tools – but not from a single interface
Difficult to manage – not user friendly
Difficult to automate
Not standard across platforms / operating systems
Although plugins will be made publically available which users can integrate with Nagios, the ISA is a self-contained, preconfigured solution, delivered within a virtual machine, that will reduce the amount of effort to implement an Ingres monitoring solution that will be supported by Ingres.
Many other open and closed source solutions are available, including: OpenNMS, Big Brother, XYMon etc
© 2010 Ingres Corporation

11. Ingres Support Appliance / Nagios
© 2010 Ingres Corporation

12. Nagios A Open Source monitoring application
GNU General Public License V3
Monitors
Hosts
Networks
Others
Monitor hub runs on Linux/Unix
Download from
© 2010 Ingres Corporation

13. Nagios Users 250,000 users worldwide 1,300 reference sites Including:
© 2010 Ingres Corporation

14. Monitors Host resources Processor load Disk usage System logs Others
Network services
SMTP, POP3, HTTP, NNTP, SNMP, FTP, SSH, ...
Just about anything else, for example
Door access alarms
Light sensors
Monitoring of network services – SMTP, POP3, HTTP, NNTP, ICMP, SNMP, FTP, SSH)
Monitoring of host resources (processor load, disk usage, system logs) on a many network operating systems
Monitoring of anything else like probes (temperature, alarms...) which have the ability to send collected data via a network to specifically written plugins
Monitoring via remotely-run scripts via Nagios Remote Plugin Executor (NRPE)
Plugins available for graphing of data (Nagiosgraph, Nagiosgrapher, PHP4Nagios etc)
Parallelized service checks available
Ability to define network host hierarchy using "parent" hosts, allowing detection of and distinction between hosts that are down and those that are unreachable
Automatic log file rotation
© 2010 Ingres Corporation

15. Nagios Monitors via plug-ins Using:
Nagios Remote Plug-in Executor (NRPE)
SSH
Monitoring of network services – SMTP, POP3, HTTP, NNTP, ICMP, SNMP, FTP, SSH)
Monitoring of host resources (processor load, disk usage, system logs) on a many network operating systems
Monitoring of anything else like probes (temperature, alarms...) which have the ability to send collected data via a network to specifically written plugins
Monitoring via remotely-run scripts via Nagios Remote Plugin Executor (NRPE)
Plugins available for graphing of data (Nagiosgraph, Nagiosgrapher, PHP4Nagios etc)
Parallelized service checks available
Ability to define network host hierarchy using "parent" hosts, allowing detection of and distinction between hosts that are down and those that are unreachable
Automatic log file rotation
© 2010 Ingres Corporation

16. Interface Notifications via E-mail Pager SMS Optional Web interface
Event handlers for proactive problem resolution
Monitoring of network services – SMTP, POP3, HTTP, NNTP, ICMP, SNMP, FTP, SSH)
Monitoring of host resources (processor load, disk usage, system logs) on a many network operating systems
Monitoring of anything else like probes (temperature, alarms...) which have the ability to send collected data via a network to specifically written plugins
Monitoring via remotely-run scripts via Nagios Remote Plugin Executor (NRPE)
Plugins available for graphing of data (Nagiosgraph, Nagiosgrapher, PHP4Nagios etc)
Parallelized service checks available
Ability to define network host hierarchy using "parent" hosts, allowing detection of and distinction between hosts that are down and those that are unreachable
Automatic log file rotation
© 2010 Ingres Corporation

17. Nagios architecture Nagios NRPE Config Files Data Files
Hub Plug-in
Monitoring Host
(Nagios Hub)
Nagios
check_nrpe
Config Files
Data
Files
Nagios Remote Program Executor
“check” scripts
Plug-in
Exit status and messages returned to Nagios
0 = OK, 1 = Warning, 2 = Critical, 3 = Unknown
Nagios requests execution of a plugin
and stored in Nagios data files
Remote (monitored)
Host
NRPE
Local
Services
check_disk
check_ingres
check_cpu
© 2010 Ingres Corporation

18. Off-the-shelf Plug-ins
~140 supported by Nagios development team
~1600 community contributions
Downloadable from
Remote monitoring supported through SSH or SSL encripted tunnels.
Simple plugin design that allows users to easily develop their own service checks depending on needs, by using the tools of choice (shell script, C++, Perl, C++, Perl, Ruby, Python, PHP, C# etc)
© 2010 Ingres Corporation

19. Custom Plug-ins Simple scripts written in Shell script C++ C# Perl
Python
PHP
Ruby
Etc.
Remote monitoring supported through SSH or SSL encrypted tunnels.
Simple plugin design that allows users to easily develop their own service checks depending on needs, by using the tools of choice (shell script, C++, Perl, C++, Perl, Ruby, Python, PHP, C# etc)
© 2010 Ingres Corporation

20. Custom Plug-ins for Ingres
To monitor server processes
For Error log mining
To measure DMF cache hit ratios
To identify
Tables in overflow
Long running queries
© 2010 Ingres Corporation

21. Guide to Building a Custom Plug-in
© 2010 Ingres Corporation

22. Guide to Building a Custom Plug-in
Step 1 - Design
Step 2 - Develop
Step 3 - Test
Step 4 - Deploy
Step 5 - Execute
Step 6 - Contribute
© 2010 Ingres Corporation

23. Step 1 - Design Check there is not a plug-in that already does this
Review development guidelines
Define plug-in specification
May include:
Triggers for exit codes
Messages to be returned to Nagios
Performance data for graphing
Persistent data for loading into Ingres tables
© 2010 Ingres Corporation

24. Step 2 - Develop Write plugin using preferred development tools
Use source code revision control
subversion
sccs
Locate scripts (or executables) in
/usr/local/nagios/libexec
© 2010 Ingres Corporation

25. Step 3 - Test For example: su - nagios cd /tmp
/usr/local/nagios/libexec/check_isa_ingres_lrq_report -s /opt/IngresII –d demodb
echo "Plugin Return Code: $?"
© 2010 Ingres Corporation

26. Step 4 - Deploy For host(s) to be monitored Copy plug-in into
/usr/local/nagios/libexec
© 2010 Ingres Corporation

27. Step 4 - Deploy
Define plugin as a command within Nagios configuration file:
/usr/local/nagios/etc/objects/commands.cfg
define command{
command_name check_remote_ingres_lrq_report
command_line $USER1$/check_by_ssh -p $ARG1$ -l nagios
–i /usr/local/nagios/etc/keys/$HOSTNAME$
-H $HOSTADDRESS$
-C /usr/local/nagios/libexec/check_isa_ingres_lrq_report -s $ARG2$ -d $ARG3$'}
© 2010 Ingres Corporation

28. Step 4 - Deploy
Place service definitions within Nagios configuration file:
/usr/local/nagios/etc/objects/services.cfg
define service{
use remote-service
host_name prod_svr
service_description demodb – Long Running Queries
check_command
check_remote_ingres_lrq_report!22!
/opt/IngresII!demodb}
© 2010 Ingres Corporation

29. Step 5 - Execute Automatically by Nagios depending on Time periods
Service checks enabled / disabled
Manually by user:
From a command line
From the Service Commands menu of the web interface
© 2010 Ingres Corporation

30. Step 6 - Contribute Contributing plug-in to the community
© 2010 Ingres Corporation

31. Example Plug-in
© 2010 Ingres Corporation

32. Long Running Queries Plug-in
Monitor plug-in
Generates a list of long running queries
Using ima
Report
A Nagios plug-in
Sends list of long running queries back to Nagios hub
Optional Ingres loader
Load list of long running queries into an Ingres table
Nagios plugins are expected to run in less than 10 seconds. By its nature, to identify LRQ requires that monitoring of queries is performed over a period of time (possibly many hours). Thus the solution implemented here is to split the “monitoring” and “reporting” elements into two distinct functions. The optional “Ingres loader” function takes the output generated by the monitor element and load the details of LRQ into an Ingres database. This data may then be used outside Nagios to generate reports, enable further analysis and provides a persistent data store (under the control of the DBA rather than Nagios).
© 2010 Ingres Corporation

33. Monitor Plug-in Shell script (~2300 code lines)
Generates a list of long running queries
For a defined time period (e.g. 8 hours)
Do the following:
Identify running queries using ima
Record details of running queries
Sleep
To reduce impact on installation being monitored only session temporary tables are used for storing data. All persistent data (for example that required between each cycle within the monitoring period) are held within flat-files (the location of which be default is /tmp, but can be defined via a con
List of long running queries includes, user name, query and approximate time taken to run. The approximation is due to; queries that may have started or continue to run outside the monitoring period, for large n second values, queries may start / finish part way through a check cycle and server performance may impact the time taken for the monitor script to run.
© 2010 Ingres Corporation

34. ima Query /* Declare a GTT to store queries currently
running against the database */
declare global temporary table session.ima_server_sessions
as select
session_id,
effective_user,
session_query
from
ima_server_sessions
where
session_query != ''
and db_name = '${h_clv_database_name}'
on commit preserve rows with norecovery \p\g
Session query currently restricted to 1000 characters
© 2010 Ingres Corporation

35. Report Shell script (~600 lines of code) Initiated by Nagios
Analyse and report on long running queries
Sends information back to Nagios Hub OK
No long running queries identified
Warning
1 or more queries took more than n seconds to run
Critical
Unable to run monitor script
© 2010 Ingres Corporation

36. Optional Ingres loader
Not a requirement for Nagios to work
Provides persistence
Takes the list of long running queries
Loads it into an Ingres table
May be used for auditing / reporting
Shell script (~800 lines of code)
Not part of the Nagios implementation, this module will store collected data in an Ingres database for subsequent reporting / analysis.
© 2010 Ingres Corporation

37. Service status details for all hosts screenshots
Dashboard: Summary of host
and service status
Service status and messages
returned from plug-in
© 2010 Ingres Corporation

38. Service status for long running queries
Status Information:
WARNING - Installation: A9, Database: ltp, Pause: 1(sec), Run Time: 5(min). Last check started 06/05/ :03:21 and completed 06/05/ :08:25. 1 query was identified: User: ingres Query: select * from bw_boat_listing, bw_boat_listing, bw_boat_listing (seen 123 times)

39. Summary Monitoring Ingres Why Monitor What to Monitor When to Monitor
Tools for Monitoring
Ingres Support Appliance / Nagios
Architecture Overview
Plug-ins
Guide to Building a Custom Plug-in
Example Plug-in
© 2010 Ingres Corporation