February 2006 Iosif Legrand 1 Iosif Legrand California Institute of Technology February 2006 February 2006 An Agent Based, Dynamic Service System to Monitor, Control and Optimize Distributed Systems Control and Optimize Distributed Systems

February 2006 Iosif Legrand 2 The MonALISA Framework   MonALISA is a Dynamic, Distributed Service System capable to collect any type of information from different systems, to analyze it in near real time and to provide support for automated control decisions and global optimization of workflows in complex grid systems.   The MonALISA system is designed as an ensemble of autonomous multi-threaded, self-describing agent-based subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks. These agents can analyze and process the information, in a distributed way, and to provide optimization decisions in large scale distributed applications.

February 2006 Iosif Legrand 3 MonALISA is A Dynamic, Distributed Service Architecture   The framework is based on a hierarchical structure of loosely coupled agents acting as distributed services which are independent & autonomous entities able to discover themselves and to cooperate using a dynamic set of proxies or self describing protocols.   An agent-based architecture provides the ability to invest the system with increasing degrees of intelligence; to reduce complexity and make global systems manageable in real time. For an effective use of distributed resources, these services provide adaptability and self-organization.

February 2006 Iosif Legrand 4 Lookup Service MonALISA service & Data Handling Data Cache Service & DB Configuration Control (SSL) Configuration Control (SSL) Lookup Service Data Stores WEB Service WSDL SOAP Client (other service) Java Discovery Registration Client (other service) Web client data Postgres MySQL Applications User defined loadable Modules to write /sent data Predicates & Agents Communications via the ML Proxy MonALSIA Service

February 2006 Iosif Legrand 5 The MonALISA Discovery System & Services Network of JINI-LUSs Secure & Public MonALISA services Proxies Clients, HL services repositories Distributed Dynamic Discovery- based on a lease Mechanism and REN Distributed System for gathering and Analyzing Information. Dynamic load balancing Scalability & Replication Security AAA for Clients Global Services or Clients Fully Distributed System with no Single Point of Failure AGENTS

February 2006 Iosif Legrand 6 Monitoring Internet2 backbone Network u Test for a Land Speed Record u ~ 7 Gb/s in a single TCP stream from Geneva to Caltech

February 2006 Iosif Legrand 7 The UltraLight Network BNL ESnet IN /OUT

February 2006 Iosif Legrand 8 Monitoring Network Topology Latency, Routers NETWORKS AS ROUTERS

February 2006 Iosif Legrand 9 Monitoring The GLORIAD Ring

February 2006 Iosif Legrand 10 Monitoring Grid sites, Running Jobs, Network Traffic, and Connectivity TOPOLOGY JOBS ACCOUNTING

February 2006 Iosif Legrand 11 Monitoring OSG: Resources, Jobs & Accounting 42 SITES ~ Nodes ( CPUs) Thousands of Jobs Thousands of Jobs parameters parameters Running Jobs Accounting

February 2006 Iosif Legrand 12 FTP Data Transfer between GRID sites Total FTP Traffic per VO

February 2006 Iosif Legrand 13 Bandwidth Challenge at SC Gbs ~ 500 TB Total in 4h

February 2006 Iosif Legrand 14 End User / Client Agent LISA- Localhost Information Service Agent   Authorization   Service discovery   Local detection of the hardware and software configuration   Complete end-system monitoring: Per-process load, I/O and network throughputs, etc.   End-to-end performance measurements   Will act as an active listener for all events related with the requests generated by its local applications.

February 2006 Iosif Legrand 15 Host Monitoring at SC2005 u Many “network” problems are actually endhost problems: misconfigured or underpowered end-systems u The LISA application was designed to monitor the endhost and its view of the network. u For SC|05 we developed we used LISA to gather the relevant host details related to network performance u Information on the system information, TCP configuration and network device setup was gathered and accessible from one site. u Future plans are to coordinate this with LISA and deploy this as part of OSG. The Tier-2 centers are a primary target. Network Device Information TCP Settings Host/System Information

February 2006 Iosif Legrand 16 Available Bandwidth Measurements Embedded Pathload module.

February 2006 Iosif Legrand 17 Coordination Service for Available Bandwidth Measurements u Enforces measurement fairness u Avoids multiple probes on shared network segments u Dynamic configuration of measurements timing u Logs events u Provides service redundancy by using a master- slave model

February 2006 Iosif Legrand 18 Monitoring the Execution of Jobs and the Time Evolution SPLIT JOBS LIFELINES for JOBS Job Job1 Job2 Job3 Job 31 Job 32 Summit a Job DAG

February 2006 Iosif Legrand 19 ApMon – Application Monitoring MonALISA Service MonALISA Service ApMon APPLICATION Monitoring Data UDP/XDR Mbps_out: 0.52 Status: reading App. Monitoring MB_inout: ApMon Config parameter1: value parameter2: value App. Monitoring... Time;IP;procID Monitoring Data UDP/XDR Monitoring Data UDP/XDR load1: 0.24 processes: 97 System Monitoring pages_in: 83 MonALISA hosts Config Servlet Library of APIs (C, C++, Java, Perl. Python) that can be used to send any information to MonALISA services   Flexibility, dynamic configuration, high communication performance dynamic reloading ApMon configuration generated automatically by a servlet / CGI script   Automated system monitoring   Accounting information No Lost Packages

February 2006 Iosif Legrand 20 Optical Switch Runs a ML Demon > ml_path IP1 IP4 “copy file IP4” ML proxy services used in Agent Communication ML Demon Control and Monitor the switch Optical Switch MonALISA ML Agent MonALISA ML Agent MonALISA ML Agent Discovery & Secure Connection 4 MonALISA agents to create on demand on an optical path or tree Time to create a path on demand <1s independent of the location and the number of connections Time to create a path on demand <1s independent of the location and the number of connections

February 2006 Iosif Legrand 21 Monitoring and Controlling Optical Planes Port power monitoring Controlling

February 2006 Iosif Legrand 22 Monitoring Optical Switches Agents to Create on Demand an Optical Path

February 2006 Iosif Legrand 23 Major Communities   OSG   CMS   ALICE   D0   STAR   VRVS   LGC RUSSIA   SE Europe GRID   APAC Grid   UNAM Grid   ABILENE   ULTRALIGHT   GLORIAD   LHC Net   RoEduNET Communities using MonALISA ABILENE VRVS - -  ALICE CMS-DC04 Demonstrated at:   SC2003   Telecom World 2003   WSIS 2003   SC 2004   I   TERENA 2005   IGrid 2005   SC 2005 MonALISA Running 24 X 7 at 250 Sites   Collecting 250,000 parameters in near real-time   Update rate of 25,000 parameter updates per second   Monitoring   12,000 computers   > 100 WAN Links   Thousands of Grid jobs running con- currently

February 2006 Iosif Legrand 24 The MonALISA Architecture Provides:  Distributed Registration and Discovery for Services and Applications.  Monitoring all aspects of complex systems :  System information for computer nodes and clusters  Network information : WAN and LAN  Monitoring the performance of Applications, Jobs or services  The End User Systems, its performance  Video streaming  Can interact with any other services to provide in near real-time customized information based on monitoring data  Secure, remote administration for services and applications  Agents to supervise applications, trigger alarms, restart or reconfigure them, and to notify other services when certain conditions are detected.  The MonALISA framework is used to develop higher level decision services, implemented as a distributed network of communicating agents, to perform global optimization tasks.  Graphical User Interfaces to visualize complex information