1. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 GridCC: Real-time Instrumentations Grids A real-time interactive GRID to integrate instruments, computational and information resources widely spread on a fast WAN Francesco Lelli Istituto Nazionale di Fisica Nucleare Laboratori Nazionali di Legnaro, Legnaro Italy

2. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Overview The GridCC Project: Introduction Bringing Instrument into the Grid: the Instrument Element The GridCC Test-bed: Pilot application Instrument Instrumentation Fast Instrument Communication Channel Standard Grid Interaction Current Implementation performance analysis

3. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 General on the GridCC Project Participant name Country Istituto Nazionale di Fisica NucleareItaly Institute Of Accelerating Systems and Applications Greece Brunel UniversityUK Consorzio Interuniversitario per Telecomunicazioni Italy Sincrotrone Trieste S.C.P.AItaly IBM (Haifa Research Lab)Israel Imperial College of Science, Technology & Medicine UK Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche Italy Universita degli Studi di UdineItaly Greek Research and Technology Network S.A. Greece It is a 3 years project. Started the 1st September 04 Funded by EU in the Frame Program 6 10 Partners from 3 EU Countries + (Israel) About 40 people engagged www.gridcc.org

4. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 The Grid Technologies to extend the limit of a single computer (center) Grid Technologies User Interface Grid Gateway ComputingElement StorageElement ComputingElement ComputingElement

5. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Extending the Grid Concepts Grid Technologie s Satellite views to monitor the volcano Control and Monitor Room To model calculations and disaster predictions Terrestrial probes to monitor The volcano activities Grid Gateway

6. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 The GridCC Project Instruments Grid Computational Grid GridCC Project + Data for Model Calculations Predictions

7. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Storage Elements Storage Elements Computing Element Computing Element Instrument Element Instrument Element: global scenario Computing Element Storage Element Instrument Element Instrument Element Existing Grid Infrastructures Web Service Interface Virtual Control Room Virtual Control Room Exec. Service WfMS WMS AgrS User direct Action Indirect Action

8. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Virtual Control Room (VCR) All end user access is via the VCR Instrument elements (IE) The IE is a virtualization of the real physical instrument Instrument elements (IE) Instrument elements (IE) Of course there may be many IEs Compute and Storage Elements (with advanced reservation) Storage Element (SE) Compute element (CE) Of course Many CEs and SEs Storage Element (SE) Compute element (CE) Storage Element (SE) Compute element (CE) Collaborative Services (CS) Virtual Control Room (VCR) Users generally not working alone Direct access to IE SE (and CE) possible but often not desirable Information and Monitoring Services (IMS) “Fast” all pervasive messaging system Information System (IS) Slowly updating information Security Services Security is essential to the success of the project Global Problem Solver Watching (via the IMS) for problems anywhere in the system and acting to resolve them. Execution Services More complex workflows, including advanced reservation and QoS guarantees, allowed The GridCC Architecture

9. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 IE Requirements Web Services Instrument Element Any Protocol or physical connection Sensor Network Instrument Grid Computing Element ElementStorage Computing InstrumentElement W E F A B C D 1: Provide a uniform access to the physical device 2: Allow a standard grid access to the instruments 3: Allow the cooperation between different instruments that belong to different VOs

10. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Element: a Black Box IE VIGS Instrument Instrumentation Fast communication channel The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. Grid Interaction Data Mover Instruments Quick Answers to the previous slide: 1)The VIGS provide the a uniform instrument instrumentation way 2)The fast communication channel disseminate the acquired information between instruments 3)The Data Mover provide a standard Grid Interface in order to be accessed by others Grids components like the SE and the CE

11. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Instrumentation

12. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Device Virtualization Model Instrument Parameters Attributes Control Model XML Based Language 1.Parameters hold configuration information 2.Attributes hold instrument variables 3.Control Model hold actions 4.XML Based Language to allow the device to describe itself Parameters: Maximum Voltage, Minimum voltage Attributes: measured Voltage Commands: Perform a measure Voltmeter

13. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Instrumentation getContexts getInstrumentManagers getInfo getIstance get/Set Parameters getCommands executeCommand getState getStateMachine IE VIGS lockInstruments unlokInstruments retrieveLoked getRemoteExecutionTime getOneWayCost getTotalMethodExecutionTime Instruments We can divide the Instrumentation in 3 main parts: The direct access to the Instruments The advance instrument reservation (interaction with the Agreement Service (AS)) in order to achieve (hard) guarantees The Possibility to predict the execution time of the instrumentation methods in a concurrent access (soft guarantees) Instrumentation method Documentation http://sadgw.lnl.infn.it:2002/IEFacade Crucial non-Functional Requirements: Instruments could be order of 10 6 Only authorized people should access to the instruments of a VO The instrumentation is not a batch process like a job submission! Interactivity is mandatory A Distribute and hierarchic implementation is mandatory the Security overhead should be negligible

14. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Element Architecture Virtual Instrument Grid Service (VIGS) Resource Service Inf & Mon Service Problem Solver Instrument Manager Instrument Element Data Mover IMS Proxy Control Manager Data Collector Real Instruments Data Flow Control Flow State Flow Error Flow Monitor Flow The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. Access Control Manager execute() getState() create() destroy() Input Manager Event Processor FSM Engine Resource Proxy Control Manager

15. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Element Implementations Resource Service Inf & Mon Service Problem Solver Instrument Manager Instrument Element Data Mover Access Control Manager The IE components are typically implemented into a fully equipped Machines (e.g. dual core cpus, large memory, large disks, etc). This is true for RS, IMS and PS. For IM (and DM) there are 2 possibilities, according to the application type: IM implemented in a fully equipped machine IM embedded into the instrument that should be controlled IM RS IMS IM Embedded Web Service

16. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Manager IM is composed by 3 main components: - Control Manager: - Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs, errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability - FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources. - Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen. - IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS IMS Proxy Data Collector Instrument Manager Input Manager Event Processor FSM Engine Resource Proxy Control Manager Instruments Data Flow State Flow Error Flow Monitor Flow Control Flow Customizable Plug-in modules to interface to the instruments

17. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Resource Service Architecture The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). A resource can be any hardware or software component involved in the IE (instruments, Instrument Managers, IMS components) RS stores the configuration data of the resources and download them to resource target when necessary Resources can be discovered, allocated and queried. It is the responsibility of the RS to check resource availability and contention with other active partitions when a resource is allocated for use. A periodic scan of the registered resources keeps the configuration database up to date. RS is interfaced to the WMS Discovery Manager Subscribe Manager Partition&Lock Manager Configuration Manager Available Resources Partition Definitions Configuration Definitions RS Data Bases Partition/Configuration retrieve methods Partition and Lock setting methods Configuration setting methods Discovery methods

18. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Information and Monitor System (IMS) PUBLISHERS (Instruments nodes) SUBSCRIBERS Errors Log info Monitor State The Information and Monitor Service (IMS) collects messages and monitor data coming from GRID resources and supporting services and stores them in a database. There are several types of messages collected from the sub-systems. The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms. The IMS collects and organizes the incoming information in a database and publishes it to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level. Instrument Manager Instruments Instrument Manager Instruments Instrument Manager Instruments

19. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Problem Solver IMS Proxy Control Manager Instrument Manager Pub/Sub IMS Proxy Control Manager Instrument Manager IMS Proxy Control Manager Instrument Manager IMS Proxy Control Manager Instrument Manager DB Data Mining Tools Algorithms evaluations : Rule Induction, Tree, Functions, Lazy, Clusters and Associative State Flow Error Flow Monitor Flow On Line Analisys Problem Solver Step 1 The control manager can perform an autonomous recovery action where the cost for the determination it is not so heavy. Step 2 Persistent information can be analyzed in order to extract knowledge Step 3 On-line information can be analyzed in order to detect possible malfunctions

20. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Poviding QoS over Web Sevices t0t0 t2t2 t1t1 t3t3 t8t8 t4t4 t7t7 t6t6 t5t5 Serialization Deserialization Transmission Processing Operation execution Client sideNetworkService side A Remote method Invocation: Avg =f(Cpu, Inputsize, Outputsize, Algorithm, Key-Factor, net) SDev =F(Cpu, Inputsize, Outputsize, Algorithm, Key-Factor, net) Cpu = machine HD + machine load (client and server side) Algorithm = method semantic Net = bandwidth + RTT Key-Factor = input value that change the method semantic Input size, Output size =effective type and dimension Crucial Times are: t3-t0 One Way Cost t4-t0 Remote Execution Cost t7-t0 Total Method Execution Cost

21. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Virtualizing Real devices linked Web Cam Position Video Streaming linked Max Value Temperature Resource Service Inf & Mon Service IE Data Mover execute() getState() create() destroy() IM Sensor Data for Model Calculations Predictions Each IM Represent the virtualization of a device IM Cam Unlinked min Value

22. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Virtualizing Real devices linked Unlinked Web Cam Position Video Streaming linked Max Value Temperature Unlinked min Value Resource Service Inf & Mon Service IM Cam IE Data Mover execute() getState() create() destroy() IM Sensor Data for Model Calculations Predictions Each IM Represent the virtualization of a device IM Master Controller

23. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Implementing Real devices (II) linked Unlinked Web Cam Position Video Streaming linked Max Value Temperature Unlinked min Value R SIMS IM Cam IE Cam Data Mover R SIMS IMSensor IE Sensor Data Mover R SIMS IM Master Controller IE Master Data Mover Data for Model Calculations Predictions Each Instrument is virtualized and a 3° IE use this others IE in order to accomplish a complex functionality

24. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Incrementing Real devices (III) linked Unlinked Web Cam Position Video Streaming linked Max Value Temperature Unlinked min Value Resource Service Inf & Mon Service IM Master Controller IE Data Mover execute() getState() create() destroy() Data for Model Calculations Predictions Cam Proxy Sensor Proxy

25. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Fast Instrument Communication Channel

26. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Message Oriented Middleware Topic A Topic B Subscribers Subscribe to a given Topic with a subscribe condition Publisher publish message in asynchronous way with a given message condition Publisher and subscribers can be part of the same program or in WAN distributed machines JMS Provide a standard set of API that standardize this communication system Many Commercial and academic implementation of this API exist in both C/C++ and Java (NaradaBrokering, Sun, IBM, SonicMQ etc etc ) In Our Case: Each instrument can be a data publisher or a data consumer For more demanding application an instrument must send/receive data in a streaming way

27. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 RMM-JMS RMM-JMS is a JMS implementation on top of our high performance Reliable Multicast Messaging (RMM) layer which provides one-to-one, one-to-many data delivery or many-to-many data exchange, in a message-oriented middleware point-to-point or publish/subscribe fashion The exceptional performance supports remote and distributed control and operation of scientific instruments such as sensors and probes Multicast transport for publish/subscribe messaging: Supporting the JMS Topic-based messaging and API, with matching done at the IP multicast level. The transport is a Nack-based reliable multicast protocol. Direct (broker- less) unicast for point-to-point messaging: JMS Queues are implemented over RMM queues. The transport is the TCP protocol. Brokered unicast transport for publish/subscribe messaging. The broker receives messages from the producer in either unicast or multicast delivery mode, and sends the messages to the subscribers in either mode broker serves as a bridge in a LAN-WAN-LAN configuration Main Contribution of IBM Haifa Research Lab (Israel)

28. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Performance: message rate – the many-to-one Blade center with 12 CPUs and 1GB Ethernet switch No message loss Total throughput: 61MBytes/sec. and 67MBytes/sec. for (a) and (b) respectively

29. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Performance: message rate – the one-to-many Blade center with 12 CPUs and 1GB Ethernet switch No message loss Peak result of over than 400000 msg/sec. was reached Rate, msg size 1 Byte 0 100000 200000 300000 400000 500000 600000 051015202530 Number of Subscribers msg/sec min Max Avg SDev Rate, msg size 1000 bytes 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 051015202530 Number of Subscribers msg/sec min Max Avg SDev

30. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Performance: round trip time (RTT, Latency) Two machines with a single publisher and a single subscriber on each one Average round trip time computed over 1000 samples

31. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Standard Grid Interaction

32. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Data Mover The task of this element is to get data from the “data collector” of the IM Data can be accessed via: –Web service interface for generic data dump (e.g. slow storage, spy stream, etc.) – grid storage element (SE) and available CEs can access to the data via an SRM Interface –Http server and TCP communication for high performance had-hoc data transfer The Data Mover exposes its methods to the IE web service and can be instrumented itself as an instrument. Instrument Resources Data Mover Data Collector IM IE Web Service Interface: get_data() SRM interface Http Server and TCP/IP raw socket Data Collector IM Data Collector IM

33. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Current IE Implementation a fist taste

34. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Manager Performances (I)

35. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Instrument Manager Performances (II) 1 2 3 1 + 2 3 1 3 1 Optimized environment IM with CMS Instruments

36. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 IMS Errors/log/states messages (xml and java objs) DB TCP/IP Pub/Sub (JMS) Web Service Interface IMS Performances IMS Proxy IMS Proxy IMS Proxy ….

37. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Main IE Pilot Applications: Power Grid Instrument Manager Instrument Element... Virtual Control Room Virtual Control Room Gas Sola r Power Grid V.O

38. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Main GridCC Pilot Applications: Control and Monitor of high energy experiments

39. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Main GridCC Pilot Applications: Control and Monitor of high energy experiments

40. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 The CMS Data Acquisition O(10 4 ) distributed Objects to – control – configure – monitor On-line diagnostics and problem solving capability Highly interactive system (human reaction time - fraction of second) World Wide distributed monitor and control 2 10 7 electronics channels 40 MHz 100 Hz

41. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 CMS Prototype

42. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 CMS Prototype: IEs at work Det 1 DAQ TTS FedBuilderRuBuilder FilterFarm Trigger TOP GTPe DAQ Detector 1 8 - GridCC middleware used for CMS MTCC (Magnet Test and Cosmic Challenge) - 11 Instrument Elements with a hierarchical topology - Instruments are in these case Linux hosts where the cms on-line software is running - More than 100 controlled hosts - 25 days to the start of the data taking ! CMS Instrument Elements DAQ IE Instrument Managers

43. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 1. Taking Control Target domain "zombies" Pirated machines Domain A Pirated machines Domain B X IDS Intrusion Detection System

44. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 A DDoS Attack Domain-wise Sensor Instrument Element Target Domain Sources of the attack Sensor Instrument Element IDS Intrusion Detection System

45. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Main GridCC Pilot Applications: Remote Operation of an Accelerator Elettra Synchrotron

46. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 The other GridCC pilot applications Meteorology (Ensemble Limited Area Forecasting) Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories Geo-hazards: Remote Operation of Geophysical Monitoring Network (see first slides) Medical Devices need a close loop between the data acquisition and the output result

47. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Conclusion The GridCC project is integrating instrument into traditional computational/storage Grids. IEs need an high interaction and interactivity between itself and the users. The GridCC IE implementation is currently installed in heterogeneous applications

48. F. Lelli, Summer School on Grid Computing, Ischia, July 20, 2006 Question? Thx for your time Acknowledgement: The GridCC project is supported under EU FP6 contract 511382. More information: www.gridcc.org On-line Demo at: http://sadgw.lnl.infn.it:2002/IEFacadehttp://sadgw.lnl.infn.it:2002/IEFacade