1. INDIANAUNIVERSITYINDIANAUNIVERSITY Grid Monitoring from a GOC perspective John Hicks HPCC Engineer Indiana University October 27, 2002 Internet2 Fall Members meeting, HENP Working Group – Los Angeles

2. INDIANAUNIVERSITYINDIANAUNIVERSITY This presentation is concerned with work being done for the iVDGL/iGOC demonstration at SuperComputing 2002. Identifying the issues NOC vs. iGOC Getting information GOC tools Overview Web site: www.igoc.iu.edu

3. INDIANAUNIVERSITYINDIANAUNIVERSITY What role should the GOC play in grid monitoring? Should the GOC just collect and publish general information about the grid status? Should the GOC collect information for trouble shooting problems? Should the GOC try to direct traffic and identify potential problems analogous to an air traffic controller (suggested by Saul Youssef, Boston University) Identifying the issues

4. INDIANAUNIVERSITYINDIANAUNIVERSITY What are some of the potential problems the GOC can help solve? Resource status and availability. Computational node Storage node Network Services (MDS) Resource availability can be determined with something as simple as a ping. Resource status depends on the measurement criteria. What is the machines current load? How much disk space is available? What is the measured network throughput between nodes? Are LDAP services available on this machine? Identifying the issues (cont.)

5. INDIANAUNIVERSITYINDIANAUNIVERSITY What information does the GOC need to help solve problems? What data needs to be gathered? Grid centric (MDS). OS centric (Ganglia, Nagios). Network centric (SNMP, other network monitoring tools). What is the data and acquisition frequency? Static (total number of nodes in a cluster). Dynamic but infrequent (number of available nodes). Dynamic and frequent (jobs running on a cluster). Realtime (available network bandwidth). Identifying the issues (cont.)

6. INDIANAUNIVERSITYINDIANAUNIVERSITY The Global NOC provides first level support for network related problems typically over networks within their domain of control. The iGOC should provide first level support for network, facility, and, infrastructure related problem not necessarily with their domain of control. The Global NOC has network engineers on staff. As far as I know, there is no such thing as a grid engineer. NOC performance monitoring usually has a demarcation point (i.e. wall jack, edge device, etc.) within a homogeneous network. GOC performance monitoring must measure end to end performance in a heterogeneous network and end node environment. The GOC must use the NOC as a resource for solving problems. NOC vs. iGOC

7. INDIANAUNIVERSITYINDIANAUNIVERSITY A key component of a successful GOC is accurate contact information. In order to solve problems or monitor resources you have to know who to talk to. We are currently collecting the following contact information from each site on the grid. High Performance Computing (HPC) contact. Principle Investigator (PI). Network person or local NOC contact. Security. Storage. System administrator. Getting information

8. INDIANAUNIVERSITYINDIANAUNIVERSITY We are using and developing the following tools to meet the GOC monitoring requirements. Nagios Ganglia LDAP tools GOC and other tools GOC tools

9. INDIANAUNIVERSITYINDIANAUNIVERSITY Nagios® is a host and service monitor designed to inform you of network problems and end system problems. Nagios provides simple ping availability of resources on the network. Nagios works with a set of “plugins” to provide local and remote host service status. Custom “plugins” are relatively easy to develop. Different methods are provided for remote resource discovery. Nagios is freely available from http://www.nagios.org.http://www.nagios.org Nagios

10. INDIANAUNIVERSITYINDIANAUNIVERSITY Currently using the following built-in Nagios plugins: check_users check_load check_disk check_procs check_mem Current Nagios plugin development: check_nagios (see if a remote Nagios is running). check_aggregate (summarize and collect the status of a group of services). Nagios

11. INDIANAUNIVERSITYINDIANAUNIVERSITY There are different ways Nagios can get information from plugins. nrpep (perl version of nrpe). check_by_ssh (passive). check_by_ssh (active). Nagios remote plugin execution (perl). Easy to use once setup. uses MD5 and TripleDES. Scales reasonably well for large number of hosts. Must have remote root access to setup. Nagios

12. INDIANAUNIVERSITYINDIANAUNIVERSITY check_by_ssh (passive). Easy to use once setup. sshd already running most places. Requires crontab entry to push data to the server. Scales reasonably well for large number of hosts. check_by_ssh (active). Easy to use once setup. sshd already running most places. Does not scale well for large number of hosts. Nagios

13. INDIANAUNIVERSITYINDIANAUNIVERSITY Current iVDGL Nagios implementation for SuperComputing demo consists of star topology. One Nagios server. Using check_by_ssh (passive). Does not scale well. Quick and dirty demo installation.tp://datatag-nagios.pi.infn.ittp://datatag-nagios.pi.infn.it Proposed persistent GOC Nagios infratructure. Run a Nagios server at the gatekeeper of each cluster. Gatekeeper Nagios only responsible for local site. Aggregate summary information and send to regional Nagios server. GOC maintains Meta Nagios with grid health status. Nagios

14. INDIANAUNIVERSITYINDIANAUNIVERSITY

15. INDIANAUNIVERSITYINDIANAUNIVERSITY

16. INDIANAUNIVERSITYINDIANAUNIVERSITY

17. INDIANAUNIVERSITYINDIANAUNIVERSITY Ganglia provides a complete pseudo real-time monitoring and execution environment. Ganglia provides a mechanism that you can not only link nodes of a cluster but an entire cluster to another cluster. Ganglia Monitoring Daemon (Gmond) is a multithreaded daemon that runs on each node that you want to monitor. Ganglia Meta Daemon (gmetad) allows you to monitor clusters. The Ganglia web front end uses PHP and RRDTool. Ganglia is freely available at http://ganglia.sourceforge.nethttp://ganglia.sourceforge.net Ganglia

18. INDIANAUNIVERSITYINDIANAUNIVERSITY Ganglia has been modified to provide VO – centric reporting. Standard Ganglia does not provide layered reporting. VO – centric Ganglia has the following features: Monitoring of host resources (processor load, memory load, disk load, etc.) Simple plugin design that allows users to easily develop their own service checks (included from the standard version) Grid and VO related sensors Publishing/Retrieving summary information to third parties Optional SSL-enabled communication (meta-daemons and web-interface) MDS interface for collecting list of reporting nodes Optional web interface for viewing current network status, notification and problem history, log file, etc. Interface with Nagios(TM)Nagios(TM) Developed by Catalin Lucian, catalind@cs.uchicago.edu – University of Chicago (http://people.cs.uchicago.edu/~cldumitr). Ganglia

19. INDIANAUNIVERSITYINDIANAUNIVERSITY

20. INDIANAUNIVERSITYINDIANAUNIVERSITY

21. INDIANAUNIVERSITYINDIANAUNIVERSITY Grid centric information can be obtained from the MDS. There are a couple of good LDAP web interfaces. LDAPExplorer, http://igloo.its.unimelb.edu.au/LDAPExplorer.http://igloo.its.unimelb.edu.au/LDAPExplorer John’s LDAP Web interface, http://ldapweb.sourceforge.net.http://ldapweb.sourceforge.net There are a number of Perl modules for LDAP, http://perl-ldap.sourceforge.net The key to extracting information is understanding the schema. Find out who is responsible for the schema and take an active role in its development. Always built dynamic search queries tools. Learn to use ldapsearch and grid-info-search. LDAP tools

22. INDIANAUNIVERSITYINDIANAUNIVERSITY GOC staff are being presented with a new set of challenges. New tools are being developed to meet these challenges. A combination of new and old tools is required to monitor and troubleshoot grid issues. Future GOC staff and “Grid Engineers” will need a broad skill set in order to be affective. There are many other grid and cluster monitoring packages: MonaLisa, GOSSIP, Gridview, etc.. There are many network monitoring packages. MRTG SNAPP and other RRDTool collectors. Netflow tools. Weather Map software. OCxMON. Pinger. GOC and other tools

23. INDIANAUNIVERSITYINDIANAUNIVERSITY Questions and discussion John Hicks Indiana University jhicks@iu.edu