1. 1 Enabling Grids for E-sciencE Joachim Flammer Service Management Configuration, service instrumentation & service management Joachim Flammer on behalf of the JRA1 Integration team

2. 2 Enabling Grids for E-sciencE Joachim Flammer Service Management Overview Comments Requirements Service interactions Common issues Implementation scenarios DM JMX example Discussion points

3. 3 Enabling Grids for E-sciencE Joachim Flammer Service Management A few comments to start … Timeline –Discussion in the area of configuration an service management is going on for several months already –If we want to have something in gLite 1.x we don’t have much time left Framework –So far discussion concentrated much on configuration, dynamic configuration, … –… however the scope is much wider and is about proper service management and instrumentation Priorities –In our opinion a proper working service management is a key corner stone of gLite –It will up to the project to decide for the priorities –Discussion started in PTF … –… but we need to have a proposal ready

4. 4 Enabling Grids for E-sciencE Joachim Flammer Service Management Requirements Do we need service instrumentation ? There has been a lot of discussion … … clear project made statement about the requirement –JRA1 / SA1 presentation in Athens –Discussions with SA1 –Discussions with user community –PTF Requirements Database

5. 5 Enabling Grids for E-sciencE Joachim Flammer Service Management PTF Requirements Database Many entries related to configuration / management / logging & error handling Entries are coming from different sides (SA1, applications, …) Some examples: –100724  Grid services should have an administrative interface for: monitoring creating alarms taking out sites/services by grid operation center etc...  They should have common, extensible, set of APIs –100727  All grid services must be able to use standardized interface for error and log messages and files. Log level has to be adaptable for e.g. different level of debugging. Minimal level has to guarantee audit trails. –100726  Grid services must enable the measurement of quantities relevant to the verification of service level agreements (SLA)

6. 6 Enabling Grids for E-sciencE Joachim Flammer Service Management Service interactions types Two types of interactions: Service interacts with system –Configuration –Logging Manager/Information system interacts with –service  getting service information (version number, uptime, …)  operations on service (start, stop, clear, …)  event notification (errors, full disks, ….) –container –system

7. 7 Enabling Grids for E-sciencE Joachim Flammer Service Management Service interaction with System Configuration –reading/writing of configuration Logging –common logging format –logging to different destinations Saving of system state –recovery after shutdown

8. 8 Enabling Grids for E-sciencE Joachim Flammer Service Management Service Information State –deployed, started, stopped –Status: running/not running Relationships –Dependencies (e.g. needs service A, B, C) –Replacement services (is duplicate of service D,E) Generic information –Name –Version –Package signature –Interface version Configuration –Values –Last updates Timing –uptime –present time Service performance metrics –Last processing time –Longest processing time Logging –Log information –Number of errors Service specific internal information –Number of used DB connections –Number of requests Available services in container / node

9. 9 Enabling Grids for E-sciencE Joachim Flammer Service Management Service operation Status operation –start, stop, pause Internal operations –pinging –Different kind of service specific operations  cleanup connections  reset  redirect log information  add user  ….

10. 10 Enabling Grids for E-sciencE Joachim Flammer Service Management Service events Service wants to inform about special events –Service started, stopped, …. –Fatal error occurred –Internal problems  disk running full  performance low  need more coffee ….

11. 11 Enabling Grids for E-sciencE Joachim Flammer Service Management The challenge … Heterogeneous system –Web services  C++  Java –Daemons Common service management interfaces –basic common set for all services –can be extended by an individual service Reuse existing capabilities – minimize impact on services –e.g. start/stop Several information sources –service –container –node Standard outside communication –standard outside SOAP interface: CIM, WBEM, WSDM, …. –self descriptive Separate management from business logic –service users are interested in service API –management application are interested in management API –separate WSDL interfaces

12. 12 Enabling Grids for E-sciencE Joachim Flammer Service Management Implementation scenario We foresee a two-fold structure –one global management service per node  gateway to the outside management  collects information from different sources Service(s) Container(s) System  directs commands to different destinations Service(s) Container(s) System –service specific parts in each service  Provides common functionality Loading of configuration Interface to management service

13. 13 Enabling Grids for E-sciencE Joachim Flammer Service Management Node system logic Tomcat gLite Service Manager Implementation scenario service logic service logic gLite management application container logic log files gLite Service Service A SOAP ? SOAP ? Service B SOAP JMX configuration WBEM/CIM/WSDM via SOAP Information system

14. 14 Enabling Grids for E-sciencE Joachim Flammer Service Management Implementation scenario Implementation scenarios –Central gLite Service Manager  Java web service  Implemented in Tomcat  JMX for internal management  Combining different sources/destinations  Outside connectors: WBEM / CIM / WSDM via SOAP … –gLite service  Java web service inside tomcat Use JMX for communication central gLite Service Manager  C++ web service (extended) ServiceConfigurator for communication with central gLite Service Manager, e.g. via SOAP  Daemons Remote/Local calls for communication with central gLite Service Manager

15. 15 Enabling Grids for E-sciencE Joachim Flammer Service Management gLiteService - configuration Example for gLiteService –Managing configuration  Load from central place  Handling of complex data type  Enable querying of configuration from outside –Example service: Datamanagement Fireman MySQL catalog (org.glite.data.catalog-service-fr-mysql)  Java webservice inside tomcat  Uses JDBC pool connection to MySQL  Sets up the JDBC connection via JNDI  Uses two classes Service class org.glite.data.catalog.service.composite.FiremanCatalogSoapBindingImpl DB connection class org.glite.data.common.helpers.DBManager

16. 16 Enabling Grids for E-sciencE Joachim Flammer Service Management Tomcat gLite Manager Implementation scenario service logic Web browser log files gLite Service Fireman Service SOAP JMX configuration HTTP Configuration MBean HTTP adapter JMX server

17. 17 Enabling Grids for E-sciencE Joachim Flammer Service Management Example configuration file <test.value description=“This is a test value.” value=“HelloWorld” type=“string”/> … <container name=“egee_db” description=“EGEE Metadata Database JNDI DataSource” type=“org.apache.commons.dbcp.BasicDataSource” jndi.reference.name=“jdbc”> <username description=“The connection username.” jmx.readable=“true” jmx.writeable=“false” type=“string” value=“egee”/> <driverClassName description=“The fully qualified Java class name of the JDBC driver.” jmx.readable=“true” jmx.writeable=“false” type=“string” value=“org.gjt.mm.mysql.Driver”/> …

18. 18 Enabling Grids for E-sciencE Joachim Flammer Service Management Turning on the central configuration Step 1: Turn on the central configuration Implement the GliteService class specifying your service name to –loads configuration from central location  here: /opt/glite/etc/config/glite-SERVICENAME.cfg.xml –makes configuration information available to  central JMX server  Data Management service

19. 19 Enabling Grids for E-sciencE Joachim Flammer Service Management Turning on the central configuration... public class FiremanCatalogSoapBindingImpl extends ServiceImpl implements FiremanCatalog { private DBManager firemanDBmanager = null;... public FiremanCatalogSoapBindingImpl(){ super(); try{ firemanDBmanager = new DBManager(“jdbc/egee_db”); } } catch (DBException e){... }... }

20. 20 Enabling Grids for E-sciencE Joachim Flammer Service Management Turning on the central configuration import org.glite.management.service.GliteService;... public class FiremanCatalogSoapBindingImpl extends ServiceImpl implements FiremanCatalog { private DBManager firemanDBmanager = null; private GliteService gLiteService; private String serviceName = “catalog-service-fr-mysql”;... public FiremanCatalogSoapBindingImpl(){ super(); gLiteService = GliteService.instance(serviceName); try{ firemanDBmanager = new DBManager(“jdbc/egee_db”); // firemanDBmanager = new DBManager(“egee_db”, serviceName); // firemanDBmanager = new DBManager(“egee_db”, gLiteService); } } catch (DBException e){... }... }

21. 21 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration Step 2: Getting the configuration information There are three possibilities 1.via JNDI 2.via JMX 3.via GliteService

22. 22 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via JNDI public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; public DBManager(String pool) throws DBException { try { m_db_pool_name = pool; Context initCtx = new InitialContext(); Context envCtx = (Context) initCtx.lookup("java:comp/env"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); } catch (NamingException e) {... }

23. 23 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via JNDI public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; public DBManager(String pool) throws DBException { try { m_db_pool_name = pool; Context initCtx = new InitialContext(); // Context envCtx = (Context) initCtx.lookup("java:comp/env"); Context envCtx = (Context) initCtx.lookup("glite"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); } catch (NamingException e) {... }

24. 24 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via JNDI Necessary adoptions: –Change of JNDI path Advantages –Only need to change JNDI path –Easy to change in between gLite & non-gLite system –Can use JNDI features: notification, … –Dynamic configuration possible if supported by application –Independent of central JMX server Disadvantages –Duplication of efforts  JMX for central & remote management  JNDI for application

25. 25 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via GliteService public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; public DBManager(String pool) throws DBException { try { m_db_pool_name = pool; Context initCtx = new InitialContext(); Context envCtx = (Context) initCtx.lookup("java:comp/env"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); } catch (Exception e) {... }

26. 26 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via GliteService import org.glite.management.service.GliteService; public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; //public DBManager(String pool) throws DBException { public DBManager(String pool, GliteService gliteService) throws DBException { try { m_db_pool_name = pool; // Context initCtx = new InitialContext(); // Context envCtx = (Context) initCtx.lookup("java:comp/env"); // m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); m_dataSource = (DataSource) gliteService.getConfigurationParameter(m_db_pool_name); } catch (Exception e) {... }

27. 27 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via GliteService Necessary adoptions: –Change JNDI lookup by GliteService function call –Pass GliteService to classes that use configuration Advantages –Easy retrieval of values –Dynamic configuration possible if supported by application –Independent of central JMX server Disadvantages –Extra features (e.g. notification) need to be implemented by hand –Duplication of efforts (JMX for central, JNDI for application)

28. 28 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via JMX import javax.management.*; public class DBManager { private MBeanServer mbs; public DBManager(String pool, String serviceName) throws DBException { m_db_pool_name = pool; try{ List srvList = MBeanServerFactory.findMBeanServer(null); ListIterator srvListItr = srvList.listIterator(); while(srvListItr.hasNext()) { MBeanServer localMbs = (MBeanServer) srvListItr.next(); if (localMbs.getDefaultDomain().compareTo("glite") == 0) mbs = localMbs; } ObjectName objectName = new ObjectName(serviceName + configurationSubMBeanPath); m_dataSource = (DataSource) mbs.getAttribute(objectName, m_db_pool_name); } catch (Exception e){

29. 29 Enabling Grids for E-sciencE Joachim Flammer Service Management Getting the configuration: via JMX Changes: –Lookup of central MBean server –Lookup of MBean instead of JNDI Object Advantages –Fully integrated solution  central management and local configuration coupled –Standard management solution –Dynamic configuration possible –Can use all JMX features (notification, old/new value..) Disadvantages –Depends on central JMX server

30. 30 Enabling Grids for E-sciencE Joachim Flammer Service Management Discussions points Feedback –Comments –Ideas Requirements from your side –What do you want to see in the management interface? –What requirements do you have? –Priorities? What do you want to see first? Discussion on implementation possibilities –Architecture –Implement GliteService as  Parent class  Add-on class –JNDI / GliteService / JMX C++ configurator –News –Feedback (Eli, …) JMX studies –News –Feedback (Paolo, …)