1. www.eu-eela.eu JRA1 – Application and Infrastructure Grid Services Francisco Brasileiro Universidade Federal de Campina Grande – UFCG (Brazil) Diego Scardaci Istituto Nazionale di Fisica Nucleare – INFN (Italy) EELA-2 1 st Conference: All Members Meeting Bogotá, 24.02.2009

2. www.eu-eela.eu Agenda –Role of the JRA1 –Services developed in the context of the activity –How to get help 2 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

3. www.eu-eela.eu JRA1 Goals –Help in fostering the sustainability of the e-Infrastructure  Making the e-Infrastructure more interesting and wide spread by increasing its reach and its usability Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 3

4. www.eu-eela.eu Increase the reach of the e-infrastructure by: –Allowing the scavenging of idle resources  Create the necessary mechanisms to allow resource centres that run the OurGrid middleware to co-exist with resource centres running gLite within the EELA platform  Provide some level of interoperation between these different kinds of resource centres and their associated applications –Allowing the execution of the grid middleware on top of platforms other than Scientific Linux  Port the gLite middleware to the Windows platform  Leveraging on the multi-platform characteristics of OurGrid Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 4

5. www.eu-eela.eu Increase the usability of the e- infrastructure by: –Developing new application-oriented grid services –Leveraging the grid services provided by the OurGrid middleware to execute bag-of-tasks jobs –Facilitating the management of resource centres Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 5

6. www.eu-eela.eu Services developed –Infrastructure-oriented  The main development here allows the co- existence of service grids (gLite-based) with opportunistic grids (OurGrid-based)  More details will be presented tomorrow –Application-oriented Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 6

7. www.eu-eela.eu Grid2Win Porting of gLite middleware to Windows Platform Dario Russo, Fabio Scibilia, Elisa Ingrà INFN –Catania 7 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

8. www.eu-eela.eu The Grid2Win project aims to create a gLite User Interface (UI) and a gLite Computing Element (CE + WNs) running on Microsoft Windows. new users come in to the grid ! windows applications can run on the grid ! Grid2Win Cygwin emulates a POSIX environment on Windows; CE based on PBS and Microsoft Windows Server 2008 8 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

9. www.eu-eela.eu Grid2Win – Main Features We have ported: Grid Security Infrastructure: Credentials management with VOMS extensions commands (voms-proxy-*) Workload Management: Job submission and output retrieving commands (glite-wms-job-*) File Transfer: Secure transfer with credential management GSIFTP (globus-url-copy) RFIO commands (rf*) Data Management: File Catalogue Browsing commands (lfc-*) and data movement & replication (lcg-*) Information System: Command lcg-infosites and existing LDAP Explorer integrated in our GUI. 9 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

10. www.eu-eela.eu Links Grid2Win on EELA-2 forge: –https://forge.eu-eela.eu/projects/grid2win/ Grid2Win on EELA-2 wiki: –https://grid.ct.infn.it/twiki/bin/view/EELA2/Grid2Win 10 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

11. www.eu-eela.eu T-GSAF Transactional Grid Storage Access Framework Salvatore Scifo Consorzio Cometa – Catania Salvo Parisi IR&T Engineering s.r.l. - Catania 11 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

12. www.eu-eela.eu GSAF as Design Help We have common features, we have common problems  we need a Design Pattern Built on top of the Grid Metadata Service and Grid Data Service –collects and implements functionalities shared among applications according to “write once use anywhere” principle –reduces the knowledge gap  hiding the complexity and the fragmentation of the several gLite Data management APIs  exposing a unified interface more near to the developer mind (design patterns) rather than the Grid stuff details (API syntaxes) –acts as a black box providing  classes and related methods for applications located above  interfaces to extend the implemented capabilities 12 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

13. www.eu-eela.eu GSAF - Data Grid Application solution GSAF is an Object Oriented Framework –Java 100% - Design Pattern based –Clear design : object modeling / functional modeling –Solves the fragmentation of DMS APIs –Solves the natural (due to OGSA) inconsistence of services –Provides ACID Transactional Model (coming soon) Road Map –RFIO java native implementation –SRM encapsulation –Grelc integration –Secure Storage integration New –LFC API –CNS API AMGA is supported only one super interface for Data Grid Application modular and extensible 13 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

14. www.eu-eela.eu 1° Use Case: DMS Web Interface 14 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

15. www.eu-eela.eu Transactional GSAF Transaction Manager –Provides developers with a Transaction Pattern  autocommit(), execute(), commit(), rollback()  performs several data manipulation in Atomic Mode –Hold Runtime Exceptions to limit Data Inconsistence –System Crash Recovery to avoid Data Loss Challenge –SOA of Grid middleware makes transactions impossible at server side –GSAF works at client side  Not full access to resources and services (locks)  Not cache mechanism to save temporary data (memory)  Not backup system to store previous data status (restore)  Not full access control to distributed resources (centralization)  ….. 15 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

16. www.eu-eela.eu State of the Art GSAF –Framework released –Adopters  ADAT Project (Archdiocese of Catania): Cultural Heritage Digital Archive (antique manuscripts)  BM Portal project (Bio-Lab, DIST University of Genoa ) :Service platform to deploy and delivery bio medical and bio informatic applications  Aiuri project (COPPE/UFRJ - BRAZIL)‏:aims to implement a Grid Oriented platform to support data and text mining applications uses GS. T-GSAF –Analysis and Design (Completed!) –Development, testing, production (Almost DONE!) 16 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

17. www.eu-eela.eu Links GSAF on EELA-2 forge: –https://forge.eu-eela.eu/projects/gsaf/ GSAF on EELA-2 wiki: –https://grid.ct.infn.it/twiki/bin/view/EELA2/GSAF 17 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

18. www.eu-eela.eu A Secure Storage Service for the gLite Middleware Diego Scardaci – INFN Catania Giordano Scuderi – UNICO S.R.L. 18 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

19. www.eu-eela.eu Insider Abuse: the problem A grid user could store sensitive data in a Storage Elements managed by external organizations. Storage Elements Administrators could access data (but the data are sensitive!). For this reason data MUST be stored in an encrypted format. Data Encryption/Decryption MUST be performed inside user secure environment (for example inside the user’s organization). 19 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

20. www.eu-eela.eu A Secure Storage service for the gLite Middleware Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way. The service is composed by the following components: Command Line Applications: commands integrated in the gLite User Interface to encrypt/upload and decrypt/ download files. Application Program Interface: allows the developer to write programs able to manage confidential data. Keystore: a new grid element used to store and retrieve the users’ keys. 20 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

21. www.eu-eela.eu Links Secure Storage on EELA-2 forge: –https://forge.eu-eela.eu/projects/securestorage/ Secure Storage on EELA-2 wiki: –https://grid.ct.infn.it/twiki/bin/view/EELA2/SecureStorage 21 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

22. www.eu-eela.eu WatchDog Monitoring and controlling job execution on the WorkerNode Riccardo Bruno INFN – Catania 22 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

23. www.eu-eela.eu Why do we need a WatchDog ? Why –Especially long term jobs require to be monitored and controlled during their execution. How –Perform job control and monitoring using grid services in the less invasive way. Observations –Almost all jobs submitted on the grid are piloted by shell scripts  Shell scripting allow to get precious info in case of faults  Shell scripting can pilot more complex batch execution –Both AMGA and SE+LFC can be used as the simplest IS on the grid.  lfc-* and lcg-* tools already available for file creation and retrieve  The latency of CLI tools for the storage is very low compared to long term jobs Requirements: –Monitor job execution watching snapshot of files produced by the job execution  File snapshot will be reported on LFC+SE or AMGA servers –It would be useful to configure the monitoring tool accordingly to the user needs  Few shell environment variables can be used to configure the watchdog tool –Control the job execution accessing directly on the WN  It is possible to send commands on the WN 23 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

24. www.eu-eela.eu How does the WatchDog work ? The Watchdog is a shell script to be included in the JDL main script. –Some watchdog features:  It starts in background before to run the long term job  The watchdog runs as long as the main job  The main script can control, stop and wait until the watchdog has finished  Easily and highly configurable and customizable  The watchdog does not compromise the CPU power of the WN Amga LFC+SE Commands or Scripts JDL WN File snapshots WN UI 24 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

25. www.eu-eela.eu Links Watchdog on EELA-2 forge: –https://forge.eu-eela.eu/projects/watchdog/ WatchDog on EELA-2 wiki: –https://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog 25 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

26. www.eu-eela.eu Other JRA1 Services GENIUS: - https://glite-tutor.ct.infn.it GFAL Java API: - https://grid.ct.infn.it/twiki/bin/view/EELA2/GFALJAVAAPI - https://grid.ct.infn.it/twiki/GFAL/ gLibrary/DRI: - https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI- UF3.pdf - https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI- CBMS08.pdf Storage Accounting (SAGE): - https://forge.eu-eela.eu/projects/sage/ All information available on the JRA1 section of the EELA-2 wiki: http://wiki.eu-eela.eu → JRA1 26 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

27. www.eu-eela.eu JRA1 Resources –JRA1 Section in EELA-2 web site  Contains a summary of the activity goals and points to the Wiki site  https://www.eu-eela.eu/ https://www.eu-eela.eu/ –JRA1 Wiki Site  Contains descriptions and links for each JRA1 services  https://grid.ct.infn.it/twiki/bin/view/EELA2/JRA1Services –EELA-Forge Site  All released software  https://forge.eu-eela.eu/ https://forge.eu-eela.eu/ –Contact us at:  Francisco Brasileiro (fubica@dsc.ufcg.edu.br)fubica@dsc.ufcg.edu.br  Diego Scardaci (diego.scardaci@ct.infn.it)diego.scardaci@ct.infn.it 27 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009