October 2006 Iosif Legrand 1 Iosif Legrand California Institute of Technology An Agent Based, Dynamic Service System to Monitor, Control and Optimize Distributed Systems Control and Optimize Distributed Systems ICFA Workshop Cracow, October 2006

October 2006 Iosif Legrand 2 The MonALISA Framework   MonALISA is a Dynamic, Distributed Service System capable to collect any type of information from different systems, to analyze it in near real time and to provide support for automated control decisions and global optimization of workflows in complex grid systems.   The MonALISA system is designed as an ensemble of autonomous multi-threaded, self-describing agent-based subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks. These agents can analyze and process the information, in a distributed way, and to provide optimization decisions in large scale distributed applications.

October 2006 Iosif Legrand 3 Regional or Global High Level Services, Repositories & Clients Secure and reliable communication Dynamic load balancing Scalability & Replication AAA for Clients Distributed Dynamic Registration and Discovery- based on a lease mechanism and remote events JINI-Lookup Services Secure & Public MonALISA services Proxies HL services Agents The MonALISA Architecture Network of Distributed System for gathering and analyzing information based on mobile agents: Customized aggregation, Triggers, Actions Fully Distributed System with no Single Point of Failure

October 2006 Iosif Legrand 4 MonALISA service & Data Handling Data Store Data Cache Service & DB Configuration Control (SSL) Predicates & Agents Data (via ML Proxy) Applications Clients or Higher Level Services WS Clients and service Web Service WSDL SOAP Lookup Service Lookup Service Registration Discovery Postgres AGENTS FILTERS / TRIGGERS Monitoring Modules Collects any type of information Dynamic Loading Push and Pull

October 2006 Iosif Legrand 5 Monitoring Internet2 backbone Network u Test for a Land Speed Record u ~ 7 Gb/s in a single TCP stream from Geneva to Caltech

October 2006 Iosif Legrand 6 The UltraLight Network BNL ESnet IN /OUT

October 2006 Iosif Legrand 7 Monitoring Network Topology Latency, Routers NETWORKS AS ROUTERS

October 2006 Iosif Legrand 8 Monitoring Grid sites, Running Jobs, Network Traffic, and Connectivity TOPOLOGY JOBS ACCOUNTING

October 2006 Iosif Legrand 9 Monitoring OSG: Resources, Jobs & Accounting 58 SITES ~ 8000 Nodes ( CPUs) Thousands of Jobs Thousands of Jobs parameters parameters Running Jobs Accounting

October 2006 Iosif Legrand 10 CMS Aggregate Job Monitoring In-depth or abstracted high-level information, as needed

October 2006 Iosif Legrand 11 ALICE : Jobs & resource usage monitoring Cumulative parameters è CPU Time è Wall time è Input & output traffic (xrootd) è Read & written files Running parameters è Resident memory è Virtual memory è Open files è Workdir size è Disk usage è CPU usage Aggregated per site

October 2006 Iosif Legrand 12 Available Bandwidth Measurements Embedded Pathload module.

October 2006 Iosif Legrand 13 FTP Data Transfer between GRID sites Total FTP Traffic per VO

October 2006 Iosif Legrand 14 ApMon – Application Monitoring MonALISA Service MonALISA Service ApMon APPLICATION MonitoringD ata UDP/XDR Mbps_out: 0.52 Status: reading App. Monitoring MB_inout: ApMon Config parameter1: value parameter2: value App. Monitoring... Time;IP;procID MonitoringD ata UDP/XDR MonitoringD ata UDP/XDR load1: 0.24 processes: 97 System Monitoring pages_in: 83 MonALISA hosts Config Servlet dynamic reloading ApMon configuration generated automatically by a servlet / CGI script No Lost Packages Lightweight library of APIs (C, C++, Java, Perl, Python) that can be used to send any information to MonALISA Services High comm. performance FlexibleAccounting Sys Mon

October 2006 Iosif Legrand 15 Monitoring the Execution of Jobs and the Time Evolution SPLIT JOBS LIFELINES for JOBS Job Job1 Job2 Job3 Job 31 Job 32 Summit a Job DAG

October 2006 Iosif Legrand 16 Long History DB The Alien Monitoring Architecture LCG Tools ApMon AliEn Job Agent ApMon AliEn Job Agent ApMon AliEn Job Agent MonALISA LCG Site ApMon AliEn CE ApMon AliEn SE ApMon Cluster Monitor ApMon AliEn TQ ApMon AliEn Job Agent ApMon AliEn Job Agent ApMon AliEn Job Agent ApMon AliEn CE ApMon AliEn SE ApMon Cluster Monitor ApMon AliEn IS ApMon AliEn Optimizers ApMon AliEn Brokers ApMon MySQL Servers ApMon CastorGrid Scripts ApMon API Services MonaLisaRepository Aggregated Data rss vsz cpu time run time job slots free space nr. of files open files Queued JobAgents cpu ksi2k job status disk used processes load net In/out jobs status sockets migrated mbytes active sessions MyProxy status

October 2006 Iosif Legrand 17 Operational Decisions and Actions  Based on monitoring information, actions can be taken in  ML Service  ML Repository  Actions can be triggered by  Values above/below given thresholds  Absence/presence of values  Correlation between multiple values  Operational actions  Alerts   Instant messaging  Supervision for Services  External commands  Event logging GlobalService ML Service Actions based on global information Actions based on local information Traffic Jobs Hosts Apps Networking Temperature Humidity A/C Power … Sensors Local decisions Global decisions ML Service

October 2006 Iosif Legrand 18 ALICE Examples MySQL daemon is automatically restarted when it runs out of memory Trigger: threshold on VSZ memory usage ALICE Production jobs queue is automatically kept full by the automatic resubmission Trigger: threshold on the number of aliprod waiting jobs Administrators are kept up-to-date on the services’ status Trigger: presence/absence of monitored information

October 2006 Iosif Legrand 19 Monitoring Video Conference System: Reflectors and Communication Topology

October 2006 Iosif Legrand 20 Monitoring and Controlling Optical Planes Port power monitoring Controlling

October 2006 Iosif Legrand 21 Monitoring Optical Switches Agents to Create on Demand an Optical Path

October 2006 Iosif Legrand 22 “On-Demand”, Dynamic Path Allocation Internet A >bbcopy A/fileX B/path/ OS path available Configuring interfaces Starting Data Transfer Monitor Control TL1 Optical Switch MonALISA Service MonALISA Distributed Service System B OS Agent Active light path Regular IP path Real time monitoring APPLICATION LISA AGENTLISA sets up - Network Interfaces - TCP stack - Kernel parameters - RoutesLISA  APPLICATION “use eth1.2, …” LISA Agent Agent DATA CREATES AN END TO END PATH < 1s Detects errors and automatically recreate the path in less than the TCP timeout path in less than the TCP timeout

October 2006 Iosif Legrand 23 Major Communities   OSG   CMS   ALICE   D0   STAR   VRVS   LGC RUSSIA   SE Europe GRID   APAC Grid   UNAM Grid (Mx)   ITU   ABILENE   ULTRALIGHT   GLORIAD   LHC Net   RoEduNET   Enlightened Communities using MonALISA - - VRVS ALICE ABILENE VRVS OSG Demonstrated at:   Telecom World   WSIS 2003   SC 2004   Internet   TERENA 2005   IGrid 2005   SC 2005   CHEP 2006   CENIC 2006 Innovation Award for High- Performance Applications MonALISA Today Running 24 X 7 at ~300 Sites   Collecting > 600,000 parameters in near real-time   Update rate of 20,000 parameter updates per second   Monitoring   12,000 computers   > 100 WAN Links   Thousands of Grid jobs running concurrently

October 2006 Iosif Legrand 24 The MonALISA Architecture Provides:  Distributed Registration and Discovery for Services and Applications.  Monitoring all aspects of complex systems :  System information for computer nodes and clusters  Network information : WAN and LAN  Monitoring the performance of Applications, Jobs or services  The End User Systems, its performance  Environment; Video streaming  Can interact with any other services to provide in near real-time customized information based on monitoring data  Secure, remote administration for services and applications  Agents to supervise applications, trigger alarms, restart or reconfigure them, and to notify other services when certain conditions are detected.  The MonALISA framework is used to develop higher level decision services, implemented as a distributed network of communicating agents, to perform global optimization tasks.  Graphical User Interfaces to visualize complex information