1. Instruments and Sensors on the Grid Gaetano Maron gaetano.maron@lnl.infn.it Istituto Nazionale di Fisica Nucleare – Laboratori di Legnaro Legnaro (PD), Italy EU FP6 Project The GridCC Project

2. G. Maron, KVI Seminar, Groningen, March 2007 2 www.gridcc.org (EU FP6 Contract 511382) GRIDCC project: goals & objectives Architecture overview Strategic targets Pilot applications Focus on the new provided capabilities  Instrument Element  RMM-JMS as fast publishing system  Virtual Control Room Conclusions Outline

3. G. Maron, KVI Seminar, Groningen, March 2007 3 www.gridcc.org (EU FP6 Contract 511382) GRIDCC project Participant nameCountry Istituto Nazionale di Fisica NucleareItaly Institute Of Accelerating Systems and ApplicationsGreece Brunel UniversityUK Consorzio Interuniversitario per TelecomunicazioniItaly Sincrotrone Trieste S.C.P.AItaly IBM (Haifa Research Lab)Israel Imperial College of Science, Technology & MedicineUK Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche Italy Universita degli Studi di UdineItaly Greek Research and Technology Network S.A.Greece Grid Enabled Remote Instrumentation with Distributed Control and Computation It is a 3-years EU FP6 project started in September 2004 Web site: www.gridcc.org

4. G. Maron, KVI Seminar, Groningen, March 2007 4 www.gridcc.org (EU FP6 Contract 511382) 1.Remote control and monitoring of complex and distributed instrumentation 2.Tight integration between instrument grid and classical computational grid 3.Human interaction with Grids via Virtual Control Room (collaborative environment) 4.Enactment of complex workflows GRIDCC: Goals & Objectives

5. G. Maron, KVI Seminar, Groningen, March 2007 5 www.gridcc.org (EU FP6 Contract 511382) Execution ServicesCollaborative Environment GRIDCC: Architecture (I) Instruments GridComputational Grid IE CE SE Instrument Element Problem Solver Instrument Manager Information & Monitoring System DATA ES VCR Workflow Service Oriented Architecture – SOA The GRIDCC services are Web Services compliant, according to the WS-I convention. Any type of client WS-I compliant, can access the GRIDCC services without any specific software library. X509 based security is used When performances are required, a kerberos based mechanism is used to access IEs.

6. G. Maron, KVI Seminar, Groningen, March 2007 6 www.gridcc.org (EU FP6 Contract 511382) GRIDCC Architecture Web Service Interface Execution Service WfMS WMS AS Instrument elements (IE) Storage Element (SE) Instrument elements (IE) Storage Element (SE) Instrument Element (IE) Storage Element (SE) Compute element (CE) Compute element (CE) Computing Element (CE) Storage Element (SE) Storage Element (SE) Storage Element (SE) Global Problem Solver Information and Monitoring Services (IMS) Information System (BDII) Security Services Virtual Control Room Virtual Control Room Collaborative Services (CS) WMS Work Management System WfMS Work Flow Mng System AS Agreement Service Broker

7. G. Maron, KVI Seminar, Groningen, March 2007 7 www.gridcc.org (EU FP6 Contract 511382) GridCC PermanentTest Bed VOMS AS, KrbServer VCR, SE, IE, BDII IE, CE/CREAM,SE, LFC WMProxy WfMS VCR, IE

8. G. Maron, KVI Seminar, Groningen, March 2007 8 www.gridcc.org (EU FP6 Contract 511382) Strategic plan: 3 main target areas (Large-scale) scientific experiments High energy particle physics (Radio-) Telescopes Remote process control Accelerator control (Tele-) Biomedicine Robotics Automotive Electronic microscopes Widely Sparse Instrumentation Power Grids Monitoring of the territory Monitoring of the sea Geo-hazard prediction Distributed laboratories Transportation monitoring Sensor network GRIDCC Middleware

9. G. Maron, KVI Seminar, Groningen, March 2007 9 www.gridcc.org (EU FP6 Contract 511382) GRIDCC pilot applications Power Grid High Energy Physics CMS High Energy Physics CMS Particle Accelerator Device Farm Geohazard Monitoring Meteorology

10. G. Maron, KVI Seminar, Groningen, March 2007 10 www.gridcc.org (EU FP6 Contract 511382) The GridCC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory. CMS Magnet Test and Cosmic Challenge (MTCC), a milestone in the CMS construction, positively carried out. Pilot applications: CMS - I Top IE CSC IE Tracker IE HCAL IE DAQ IE RPC IE Trigger IE DT IE ECAL IE DAQ IM FBRBFF xdaq DAQ RS DAQ IMS User Interface CMS Control Structure CMS Detector

11. G. Maron, KVI Seminar, Groningen, March 2007 11 www.gridcc.org (EU FP6 Contract 511382) Pilot applications: Power grid Instrument Manager Instrument Element... Virtual Control Room Virtual Control Room Gas Solar Panel Power Grid V.O. GRIDCC deployed to monitor: A 50kW generator A 1 kw Photo-Voltaic array Power Grid. GRIDCC deployed to monitor a 50kW generator and a 1 kW Photo-Voltaic array

12. G. Maron, KVI Seminar, Groningen, March 2007 12 www.gridcc.org (EU FP6 Contract 511382) Pilot applications: Remote Operation of an Accelerator Elettra Synchrotron

13. G. Maron, KVI Seminar, Groningen, March 2007 13 www.gridcc.org (EU FP6 Contract 511382) Meteorology (Ensemble Limited Area Forecasting)  Weather forecasting system to detect hazardous weather Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories  The Device Farm consists of a pool of Measurement Instruments for Telecommunication Experiments Geo-hazards: Remote Operation of Geophysical Monitoring Network  The monitoring net will be characterized by different levels of activity: stand-by, pre-alert, alert, plus a control modality  An event worth to be monitored is for example the evolution of a “landslip” GRIDCC: other applications

14. G. Maron, KVI Seminar, Groningen, March 2007 14 www.gridcc.org (EU FP6 Contract 511382) Instrument Element  Complete integration of Instruments into SOA. Instruments become “services” accessible via standard Web Service  Complete integration into the computational and storage GRID with the embedded Sorage Element (SE)  High virtualization and abstraction of diverse physical devices  A very scalable and platform-independent framework (Java based) Fast Data Publishing System  High-performance reliable multicast via RMM-JMS Virtual Control Room  Allows effective human interactions with the grid  Provides a homogeneous collaborative environment Focus on the new capabilities provided by GRIDCC

15. G. Maron, KVI Seminar, Groningen, March 2007 15 www.gridcc.org (EU FP6 Contract 511382) The IE permits  SOA accessible operations to control and monitor the instruments (via VIGS), such as: execute a command get / set parameters  different data outputs: data mover to/from a grid Storage Element (via SRM) high bandwidth channel for data publishing (via IMS). low bandwidth channel for logs, states etc. (via IMS). Instrument Element: Versatile I/O multichannel VIGS Commands Status Data Subscribers Grid Interaction SRM VCR Storage Element (SE) Storage Element (SE) Storage Element (SE) Logs, Errors, States, Monitors Instruments IMS Parameters ES VIGSVirtual Instrument Grid Service IMSInformation and Monitor Service SRMStorage Resource Management IE

16. G. Maron, KVI Seminar, Groningen, March 2007 16 www.gridcc.org (EU FP6 Contract 511382) The Instrument Element (IE) is one of the novel elements of the GRIDCC architecture. It offers a standard web service interface to integrate scientific and general purpose instruments and sensors within the grid. An IE can control a set of instruments with the possibility to decide on which sub- set to operate. The IE is the key decoupling element between real instruments and the grid Instrument Element: Instruments as Services Web Services Instrument Element Any Protocol or physical connection Sensor Network Instrument Service Oriented Access Instrumentation

17. G. Maron, KVI Seminar, Groningen, March 2007 17 www.gridcc.org (EU FP6 Contract 511382) Instrument Element: Instrument Virtualization Instrument Parameters Attributes XML Based Language Control Mode : Perform a measure The IE grid instruments representation is basically based on:  providing the XML description of the instruments managed by the IE  providing the Finite State Machine that defines the dynamic behaviour of the control performed on the set of instruments managed by the IE  simple grid accessible operations to control and monitor the instruments like execute a command, get or set a parameter or a set of parameters of the instrument (VIGS).  providing different data acquisition output paths ranging from the accessing of a Grid Storage Element (SRM/SE) to a very efficient communication channel when an high bandwidth output stream is required (via IMS). Instruments VIGS SE IE IMS

18. G. Maron, KVI Seminar, Groningen, March 2007 18 www.gridcc.org (EU FP6 Contract 511382) VIGS IMS Proxy Control Manager Data Collector Real Instruments Data Flow Control Flow State Flow Error Flow Monitor Flow The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. Input Manager Event Processor FSM Engine Resource Proxy Control Manager IMS Instrument Element Architecture Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Instrument Protocols

19. G. Maron, KVI Seminar, Groningen, March 2007 19 www.gridcc.org (EU FP6 Contract 511382) Instrument Element Framework Characteristics The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled. Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments. the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM). relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP). Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS). IE Resource Service IMS Local Problem Solver Instrument Manager Access Control Manager Data Mover Instrument Manager VIGS Instrument IMS Proxy Control Manager Data Collector Input Manager Event Processor FSM Engine Resource Proxy Control Manager

20. G. Maron, KVI Seminar, Groningen, March 2007 20 www.gridcc.org (EU FP6 Contract 511382) VIGS IMS Instrument Manager Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Instrument Manager IMS Proxy Control Manager Data Collector

21. G. Maron, KVI Seminar, Groningen, March 2007 21 www.gridcc.org (EU FP6 Contract 511382) 21 Instrument Manager IM is composed by 3 main components: - Control Manager: - Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs,errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability - FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources. - Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen. - IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS IMS Proxy Data Collector Instrument Manager Input Manager Event Processor FSM Engine Resource Proxy Control Manager Instruments Data Flow State Flow Error Flow Monitor Flow Control Flow Customizable Plug-in modules to interface to the instruments

22. G. Maron, KVI Seminar, Groningen, March 2007 22 www.gridcc.org (EU FP6 Contract 511382) Status data collection data publishing on RMM-JMS data on file in progress data on DB to be started sync commands (enable/disable done trough the “set param” of the IE façade (VIGS) Data Collector Resource Service IMS Local Problem Solver Instrument Manager Instrument Element IMS Proxy Control Manager Data Collector Access Control Manager Input Manager Event Processor FSM Engine Resource Proxy Control Manager Data Mover VIGS IE File System RMM-JMS DB persistency enable/disable data stream Instrument data stream Data Subscribers (Monitor, data analysis, etc.)

23. G. Maron, KVI Seminar, Groningen, March 2007 23 www.gridcc.org (EU FP6 Contract 511382) VIGS IMS Proxy Control Manager Data Collector IMS Resource Service Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Resource Service

24. G. Maron, KVI Seminar, Groningen, March 2007 24 www.gridcc.org (EU FP6 Contract 511382) Resource Service Architecture The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). A resource can be any hardware or software component involved in the IE (instruments, Instrument Managers, IMS components) RS stores the configuration data of the resources and download them to resource target when necessary Resources can be discovered, allocated and queried. It is the responsibility of the RS to check resource availability and contention with other active partitions when a resource is allocated for use. A periodic scan of the registered resources keeps the configuration database up to date. Discovery Manager Subscribe Manager Partition&Lock Manager Configuration Manager Available Resources Partition Definitions Configuration Definitions RS Data Bases Partition/Configuration retrieve methods Partition and Lock setting methods Configuration setting methods Discovery methods

25. G. Maron, KVI Seminar, Groningen, March 2007 25 www.gridcc.org (EU FP6 Contract 511382) Technology used http://www.jxta.org/http://www.jxta.org/ Resources Discovery

26. G. Maron, KVI Seminar, Groningen, March 2007 26 www.gridcc.org (EU FP6 Contract 511382) VIGS Control Manager Data Collector IMS Information and Monitor Service Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Proxy

27. G. Maron, KVI Seminar, Groningen, March 2007 27 www.gridcc.org (EU FP6 Contract 511382) 27 Information and Monitor System (IMS) The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs. The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms. The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level. IMS Proxy IMS Proxy IMS Proxy PERSISTENCYPublish / Subscrib SUBSCRIBERS

28. G. Maron, KVI Seminar, Groningen, March 2007 28 www.gridcc.org (EU FP6 Contract 511382) IMS Fast Data Publishing: RMM-JMS IE IE Data ProducerData Subscribers Same data are sent to several subscribers. Multicast protocols can have a benefic impact on performances JMS provides a standard set of APIs for the communication Many commercial and academic JMS implementations both in C/C++ and Java (NaradaBrokering, Sun, IBM) GRIDCC (IBM Haifa lab) has implemented a Reliable Multicast protocol (RMM) JMS compliant RMM-JMS works within a LAN but an efficient bridge technology has been developed to allow inter-LAN multicast communication

29. G. Maron, KVI Seminar, Groningen, March 2007 29 www.gridcc.org (EU FP6 Contract 511382) VIGS IMS Proxy Control Manager Data Collector IMS Problem Solver Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Problem Solver

30. G. Maron, KVI Seminar, Groningen, March 2007 30 www.gridcc.org (EU FP6 Contract 511382) Problem Solver IMS Proxy Control Manager Instrument Manager Pub/Sub IMS Proxy Control Manager Instrument Manager IMS Proxy Control Manager Instrument Manager IMS Proxy Control Manager Instrument Manager DB Data Mining Tools Algorithms evaluations : Rule Induction, Tree, Functions, Lazy, Clusters and Associative State Flow Error Flow Monitor Flow On Line Analisys Problem Solver Step 1 The control manager can perform an autonomous recovery action where the cost for the determination it is not so heavy. Step 2 Persistent information can be analyzed in order to extract knowledge Step 3 On-line information can be analyzed in order to detect possible malfunctions

31. G. Maron, KVI Seminar, Groningen, March 2007 31 www.gridcc.org (EU FP6 Contract 511382) Status: installed and running SRM 2.2 STORM implemetation (INFN) Bestman implementation (Berkeley)) XFS IE file system Embedded SRM compliant Grid Storage Element (SE) Resource Service IMS Local Problem Solver Instrument Manager Instrument Element Access Control Manager SE Backend VIGS IE File System SRM GRID accessibile data

32. G. Maron, KVI Seminar, Groningen, March 2007 32 www.gridcc.org (EU FP6 Contract 511382) Instrument Element: Scalable on embedded systems FPGA PPC 405 Xilink Virtex IV Custom Board Custom Electronics 1 Gbps Ethernet Web Service GridCC IM JavaVM Linux USER INTERFACE Custom Logic Montavista www.montavista.com JamVM http://jamvm.sourceforge.net Standalone Axis Instrument Manager on a chip … … towards a grid on a chip IE Instrument Manager

33. G. Maron, KVI Seminar, Groningen, March 2007 33 www.gridcc.org (EU FP6 Contract 511382) IE technologies Web Service compliant (WS-I) Tomcat + Axis (and Java) and Axis standalone are the main technologies of the IE All the services are deployed on a single or multiple instances of Tomcat, according to the needs of the application Message oriented middleware (Pub/Sub) is based on the Java Messaging System (JMS). The following implementations are used in the project  Sun  Narada Brokering  RMM - JMS (GridCC IBM) MySQL and Oracle are used as Data Base for the RS SRM interface version 2.2 used

34. G. Maron, KVI Seminar, Groningen, March 2007 34 www.gridcc.org (EU FP6 Contract 511382) The VCR is a collaborative web portal Enable the user to run, aggregate and display plug-ins that act as “mini applications” Virtual Control Room

35. G. Maron, KVI Seminar, Groningen, March 2007 35 www.gridcc.org (EU FP6 Contract 511382) Conclusion - I The GridCC project is integrating instruments into the “classic” computational/storage Grids. Novel concepts introduced by GridCC are:  The Instrument Element, allowing a virtualisation of the instruments to be controlled and their insertion in a Grid  The Virtual Control Room, providing an highly interactive environment with IEs/CEs/SEs. VCR even provides cooperative tools to allow the cooperation (logbook, chat, videoconf., etc.) between remote users  hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times  Fast Data Publication via a Message Oriented middleware (RMM-JMS) to distribute data and information from an IE to the world wide Grid. The IE is highly customizable and can be adapted in different environments.

36. G. Maron, KVI Seminar, Groningen, March 2007 36 www.gridcc.org (EU FP6 Contract 511382) Conclusion - II The IE can be shrunk down into a chip allowing grid enabled embedded control of the instrumentation Several heterogeneous pilot applications are deploying and running the IE middleware We support and encourage the adoption of our middleware in other projects/experiments.

37. G. Maron, KVI Seminar, Groningen, March 2007 37 www.gridcc.org (EU FP6 Contract 511382) Any Questions? Thank you for your attention

38. G. Maron, KVI Seminar, Groningen, March 2007 38 www.gridcc.org (EU FP6 Contract 511382) Spare

39. Tecnologie usate http://www.jxta.org/http://www.jxta.org/ http://gladgw:2002/InstrumentElementKeeper/ http://gladgw:2002/InstrumentElementKeeper/ da dove vedere la lista degli strumenti http://gladgw.lnl.infn.it:2002/SeedsFinder/http://gladgw.lnl.infn.it:2002/SeedsFinder/ da dove scaricare l’IE da web-start

40. G. Maron, KVI Seminar, Groningen, March 2007 40 www.gridcc.org (EU FP6 Contract 511382) GridCC Scalability Demo Imperial Brunel Elettra IBM CNIT IASA IMAA GRNet UniUD INFN VCR Monitor GUI Network Information Provider Instrument Element Network Core Machine

41. G. Maron, KVI Seminar, Groningen, March 2007 41 www.gridcc.org (EU FP6 Contract 511382)

42. G. Maron, KVI Seminar, Groningen, March 2007 42 www.gridcc.org (EU FP6 Contract 511382) Pilot applications: CMS - II CMS Detector 2 10 7 electronics channels 40 MHz 100 Hz O(10 4 ) distributed Objects to be  controlled  configured  monitored On-line diagnostics and problem solving capability Highly interactive system (human reaction time - fractions of second) World Wide distributed monitor and control

43. G. Maron, KVI Seminar, Groningen, March 2007 43 www.gridcc.org (EU FP6 Contract 511382) Instrument Element – Interconnections II IE VCR Control Panel Computing Element (CE) Fast Data Display Fast Data Publishing Information and Monitor System Log Display Control and Status Existing Grid Elements Log Persistency Data Consumer WorkFlow Mng. System Storage Element (SE)

44. G. Maron, KVI Seminar, Groningen, March 2007 44 www.gridcc.org (EU FP6 Contract 511382) - WS-I, http://www.ws-i.org - Java Message Service (JMS), http://java.sun.com/products/jms - RMM-JSM, http://www.haifa.ibm.com/projects/software/rmsdk/gridcc.html - gLite, http://glite.web.cern.ch/glite/ - SRM, http://sdm.lbl.gov/srm-wg - StoRM, http://grid-it.cnaf.infn.it/storm - Montavista, http://www.montavista.com - JamVM, http://jamvm.sourceforge.net References

45. G. Maron, KVI Seminar, Groningen, March 2007 45 www.gridcc.org (EU FP6 Contract 511382) The INFN team working in Legnaro, in collaboration with CERN at Geneva: E. Frizziero M. Gulmini F. Lelli G. Maron A. Petrucci S. Traldi Acknowledgments

46. G. Maron, KVI Seminar, Groningen, March 2007 46 www.gridcc.org (EU FP6 Contract 511382) Instrument Element: Versatile I/O multichannel VIGS Commands Status Data Subscribers Grid Interaction SE 1)A Virtual Instrument Grid Service (VIGS) interface has been defined. The VIGS provides a Web Service acccess to the instrumentation 2)The Data Publishing (DP) channel disseminates the acquired information to the data subscribers 3) A SRM/SE interface is provided to allow file movement with other Grid Element, like SE, CE and other IEs. 4)Information messages (logs, errors, etc.) are published from the IE (via IMS) Virtual Control Room Execution Service WfMS WMS AS Compute element (CE) Compute element (CE) Computing Element (CE) Storage Element (SE) Storage Element (SE) Storage Element (SE) IE Logs, Errors, States, Monitors IMS DP VIGS Virtual Instrument Grid Service IMS Information and Monitor Service SE Storage Element DP Data Publishing

47. G. Maron, KVI Seminar, Groningen, March 2007 47 www.gridcc.org (EU FP6 Contract 511382) Fast Data Publishing: Some results 32 Dual Xeon 2.4GHz 1.5GB RAM machines, 1 GB Ethernet switch At most 1 publisher, subscriber, or broker- (Sun MQ3.6) per machine No message lost RMM throughput: 75-90 Mbytes/sec. (for 5 and more publishers) Message Rate: Case many-to-one

48. G. Maron, KVI Seminar, Groningen, March 2007 48 www.gridcc.org (EU FP6 Contract 511382) Architecture: New “instruments grid” Component NameDescription Instrument Element (IE) This is a unique concept to GRIDCC. It consists of a coherent collection of services which provide all the functionalities to configure, partition and control the physical instruments Information & Monitor Service (IMS) It gathers from GRIDCC resources information and monitor data to be disseminated through a publish / subscribe systems or to be stored in persistent repositories. Problem Solver (PS) It offers automated problem solving in a Grid environment at two levels. A local PS, within a given Instrument Element, allows to solve local problems related to functionalities of a given instrument. A global PS, allows to solve system-wide problems. Virtual Control Room (VCR) It provides a common set of collaboration tools and allows users to build complex workflows, which are then submitted to the Execution Services, and to directly monitor and control remote instruments in real-time. Execution Services (ES) They control the execution of the workflows defined by the user in the VCR, maintaining the status of the tasks that make up the workflow. They also support the advance reservation of resources. Security Services (SS) GRIDCC uses a split security system. When interacting with components of other Grids the GSI security will be used and the users identified by their X.509 proxy certificate. When interacting with the IE the user will be identified by a Kerberos ticket.