Camilo Lara KIP HLT Production Readiness Review 1 HLT Cluster Management

Camilo Lara KIP HLT Production Readiness Review 2 HLT Cluster Management Architecture Network GUI Monitoring Server SysMES Framework Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client … GUI Monitoring Server

Camilo Lara KIP HLT Production Readiness Review 3 HLT Cluster Management Architecture Network GUI Monitoring Server SysMES Framework Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client … GUI Monitoring Server Part 1 Monitoring and Visualization

Camilo Lara KIP HLT Production Readiness Review 4 HLT Cluster Management Architecture Network GUI Monitoring Server SysMES Framework Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client … GUI Monitoring Server Part 2 System Management of the cluster nodes and applications

Camilo Lara KIP HLT Production Readiness Review 5 HLT Cluster Management Architecture Network GUI Monitoring Server SysMES Framework Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client … GUI Monitoring Server Monitoring using SysMES monitors, Epics, Lemon, Ganglia - Hard disk partition usage - Avg CPU usage - System up time - Network monitoring - Process monitoring (e.g. Analysis Tasks)

Camilo Lara KIP HLT Production Readiness Review 6 HLT Cluster Management Architecture Network GUI Monitoring Server SysMES Framework Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client Cluster Node Monitoring Engine SysMES Client … GUI Monitoring Server SysMES Functionality - Object orientated modelling - Message generation and Handling - Job Management - Configuration Management - Complex Rule Handling

Camilo Lara KIP HLT Production Readiness Review 7 Part 1: Monitoring and Visualization Monitoring System Network Node Event Checking SysMES Client Interface SysMES Framework GUI Admin Computer

Camilo Lara KIP HLT Production Readiness Review 8 EPICS and SysMES - Architecture MEDM Network Node Admin Computer Event Checking SysMES Client Interface SysMES Framework Record Inp: SNMP CA Server SNMP Agent MIB devSNMP CA Client Monitoring SysMES Client Interface CA Client

Camilo Lara KIP HLT Production Readiness Review 9 LEMON and SysMES - Architecture Network Node Event Checking SysMES Client Interface SysMES Framework LEMON Agent UDP Client Sensors UDP Server Monitoring LEMON GUI Admin Computer SysMES Client Interface UDP Server

Camilo Lara KIP HLT Production Readiness Review 10 Ganglia

Camilo Lara KIP HLT Production Readiness Review 11 Part 2: System Management using SysMES System Management for Embedded Systems

Camilo Lara KIP HLT Production Readiness Review 12 System properties  Based on available, established standards  Interoperability and manufacturer independance XML (Extensible Markup Language) CIM (Common Information Model)  Object-orientated modelling of the complete system  Simple modelling of their relationships  Reusability  Decentralization  Decentralized modelling  Decentralized storage of information  Decentralized and dynamic configurations management  Decentralized management

Camilo Lara KIP HLT Production Readiness Review 13 System properties  High availability and scalability  No single-point-of-failure  Redundant storage of information  Clustering of management resources and DB  Load balancing  Flexibility  Platform independent  Self management  Automatic reaction to triggering conditions  Reliability  Transaction-based communication to avoid information loss

Camilo Lara KIP HLT Production Readiness Review 14 System functionality  Modelling  Object orientated modelling of resources in UML  Creation of Objects from this model  Transfer of Objects to management framework

Camilo Lara KIP HLT Production Readiness Review 15 System functionality  Message generation  Evaluation of the measured values in Client Dynamic decision of which values have to be processed Prevention of management environment system overload  Message Handling  Decentralized storage of messages in DB and on Client  Job Management  Communication with Clients through Jobs  Different Jobs types Configuration Jobs: e.g. changeMonitor Update Jobs: e.g. addMonitor Management Jobs: e.g. deleteMessages

Camilo Lara KIP HLT Production Readiness Review 16 System functionality  Configuration Management  Client knows its current configuration state  Client stores its current configuration for recovery  All possible configurations are stored on server  Complex Rule Handling  3 tier Rule management System  Tier 1. Rule management on the client (reaction < 10 ms)  Tier 2. Simple rule management on the server (reaction < 300 ms)  Tier 3. Complex rule management on the server using a expert system (reaction < 1 s)

Camilo Lara KIP HLT Production Readiness Review 17 Complex Rule Handling 3rd Tier Expert System  Specification: Object-oriented mapping of context-free grammar enables specification of ComplexRules ComplexRule = Composite Messages -> Action  Composite Messages = Correlation of Primitive Messages Operators for correlating messages: Basic: AND, OR, NOT, SEQ Set Oriented: AVERAGE, COUNT Operators for computing message attributes: Mathematical functional operators: (+, -, /, *, etc) Mathematical relational operators: (>, <, =, !=, etc) Boolean operators: (&&, ||, !, etc)  Action Multiple actions specifiable Actions can be parameterized with attributes of the triggering composite messages  Rules can be deployed/activated and removed/pacified on-the-fly

Camilo Lara KIP HLT Production Readiness Review 18 Complex Rule Handling - Architecture  Distributed detection of composite messages:  Several detection servers establish a detection hierarchy  Detection of composite events is performed as close as possible to the client  Servers residing on lower levels of the hierarchy inform higher level servers of composite messages occurrences  Benefits:early detection -> faster reactions early filtering -> reduced network traffic distributed detection -> lower load on each server  Clustering detection hierarchy nodes:  Avoidance of SPoF on the detection servers of a detection hierarchy by: Fail-Over node for each detection Server ( Master/Slave, Master/Master) Operation of M/S or M/M is backed by clustered pre-evaluators which perform pre-evaluations and filtering of incoming events

Camilo Lara KIP HLT Production Readiness Review 19 Complex Rules - Samples  E1 (E1.Temperature>70) AND E2 (E2.RPM>300) (E1.DeviceID = E2.DeviceID) -> deployJob(sendMail(“Lindenstruth”))  If at least 20 different machines have an current CPU temperature > 75 C call 911.  DistinctCount (E1 (E1.name=’CPU_temp’ & E1.temp>75), E2 (E2.name=’CPU_temp’& E2.temp deployJob(call(“911”))

Camilo Lara KIP HLT Production Readiness Review 20 Architecture – Physical View AccessPoint

Camilo Lara KIP HLT Production Readiness Review 21 Architecture – Logic View Deploy Module Object Management Model XMI2MOF Class Management Model Message Module Database Job Module Connection Module Rule Module CIMConnector Monitoring Message Handling Job Handling Third party Interface Operating System Rule Handling http xml mof object xmi Modelling Clients Management Framework ArgoUML / Poseidon, XMI2MOF -CIM / CIM Object Manager -CIMNavigator -EJB -JBOSS -MySQL Cluster -Java Interpreter / C Compiler -Linux

Camilo Lara KIP HLT Production Readiness Review 22 SysMES Applications - ToDo  Building a operational concept  Defining the specific objects and configuration for the SysMES Framework  Resource Manager  1. Proposal on the CIM and SBLIME (Standards Based Linux Instrumentation for Manageability)  Inventory Manager  Module for the visualization of the SysMES configuration jobs  Dynamic HLT Cluster Configuration  SysMES provides the global state of the cluster  Reconfiguration of analysis chain using configuration jobs