HEPSYSMAN Monitoring Workshop Introduction to the Day and Overview of Ganglia Pete Gronbech

Slide 2 31 st October 2007 Introduction & Ganglia Agenda Wednesday 31st October :00Start / Coffee 10: :00Introduction & GangliaOverviewPete Gronbech 11: :30MonAMI Interactive WorkshopPaul Millar 12: :30Lunch 13: :00Intro To NagiosA. Elwell 14: :30GRID Service Monitoring GroupIan Neilson 14: :00Further Nagios Scripts.Chris Brew 15: :00Live Install at a site and workshop discussion. 16: :30Other Monitoring Tools Discussion (Pakiti, gridmap, accounting cpu and storage, SAM, SAM admins page etc.) 16:30AOB and wrap up.

Slide 3 31 st October 2007 Introduction & Ganglia Why Monitoring Untrustworthy machines, that are critical. Your systems will fail. When they do fail, two things save you from downtime: Redundancy and Monitoring systems Limited Man Power at sites Ever increasing sizes of clusters Complex software with many failure modes Need to meet SLAs – 95% uptime PR and reporting

Slide 4 31 st October 2007 Introduction & Ganglia Many external monitoring sites Gstat - Steve Lloyds Page SAM -

Slide 5 31 st October 2007 Introduction & Ganglia Many more external monitoring sites Gridmap - Gridview Accounting -

Slide 6 31 st October 2007 Introduction & Ganglia Local Site Monitoring Unix tools Batch System tools Really need something that can provide a quick visual overview of the health and load on your cluster … ganglia

Slide 7 31 st October 2007 Introduction & Ganglia Ganglia

Slide 8 31 st October 2007 Introduction & Ganglia How does Ganglia work? Ganglia works through a small agent, gmond, on each node or machine to be monitored. You can distribute a single gmond instance to lots of machines at once. Gmonds communicate the state of their local node to a machine running a Master gmetad instance. The server uses RRDtool to store the data over time The Ganglia framework can be extended to monitor many parameters.

Slide 9 31 st October 2007 Introduction & Ganglia Setup The software can be downloaded from Computer A Runs gmond Computer BComputer CComputer D Runs gmond Runs gmetad Clients just have to run gmond, which is configured by /etc/gmond.conf Server to collect the data runs gmetad. It could also run gmond to monitor itself. The web interface needs to run on a webserver. Computer D gmetad gmond httpd

Slide st October 2007 Introduction & Ganglia Client Setup Computer A Runs gmond Computer BComputer C Runs gmond yum install ganglia-gmond edit config file service gmond start chkconfig gmond on /etc/gmond.conf extracts cluster { name = "LCG Workers" } /* Feel free to specify as many udp_send_channels as you like. Gmond used to only support having a single channel */ udp_send_channel { mcast_join = port = 8649 } /* You can specify as many udp_recv_channels as you like as well. */ udp_recv_channel { mcast_join = port = 8649 bind = } /* You can specify as many tcp_accept_channels as you like to share an xml description of the state of the cluster */ tcp_accept_channel { port = 8649 } udp_send_channel { port = 8649 host = pplxconfig }

Slide st October 2007 Introduction & Ganglia Server Setup yum install ganglia-gmond ganglia-gmetad ganglia-web edit /etc/gmond.conf edit /etc/gmetad.conf Computer D gmetad gmond httpd Extracts from /etc/gmetad.conf data_source "LCG Workers" computerA.physics.ox.ac.uk ComputerB.physics.ox.ac.uk computerC.physics.ox.ac.uk data_source "LCG Servers" t2se01.physics.ox.ac.uk:8656 t2ce02.physics.ox.ac.uk:8656 gridlogger.physics.ox.ac.uk:8656

Slide st October 2007 Introduction & Ganglia Aggregating sub clusters

Slide st October 2007 Introduction & Ganglia Host level detail

Slide st October 2007 Introduction & Ganglia Customizing Adding PBS Batch Queue data

Slide st October 2007 Introduction & Ganglia PBS Queue Monitoring Originally based on RAL Tier 1 work Actually fairly complicated. see Chris Brew or me later for details.

Slide st October 2007 Introduction & Ganglia How is Ganglia different from Nagios Ganglia is architecturally designed to perform efficiently in very large monitoring environments: each Ganglia gmond performs its service checks locally, reporting in at a regular interval to the gmetad. Nagios performs its service checks by polling each device across a network connection and waiting for a response (known as "active checks"), which can be more resource and bandwidth intensive. Nagios uses the results of its active checks to determine state by comparing the metrics it polls to thresholds. These state changes can in turn be used to generate notifications and customizable corrective actions. Ganglia, by contrast, has no built-in thresholds, and so does not generate events or notifications. The general rule of thumb has been: if you need to monitor a limited number of aspects of a large number of identical devices, use Ganglia; if you want to monitor lots of aspects of a smaller number of different devices, use Nagios. But those distinctions are blurring as Ganglia supports more and more devices, and as Nagios' scalability improves.

Slide st October 2007 Introduction & Ganglia How is Ganglia different from Nagios The problem with ganglia and all the other external web pages we have been looking at is that you have to look at them! If all is well with your system you dont want to have to look. This is where Nagios comes in. It can be setup to alert you when something goes wrong, or a value passes a threshold.