1. www.eu-eela.org E-science grid facility for Europe and Latin America EELA-2 JRA1 Services Diego Scardaci INFN (Italy) Joint EELA-2/EGEE-III Tutorial for Trainers Catania (Italy), 01.07.2008

2. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 2 Outline NA3/JRA1 Synergies EELA-2 Services already available: –GENIUS –GFAL Java API –gLibrary/DRI –Grid2Win –Secure Storage –Storage Accounting (SAGE) –Transactional Grid Storage Access Framework (T-GSAF) –Watchdog All information available on the JRA1 section of the EELA-2 wiki: http://wiki.eu-eela.eu → JRA1

3. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 3 Grid Services Developing Develop Services that will decisively contribute to a durable, versatile and cost- effective use of the e- Infrastructure Applications Support Provides support to EELA-2 Applications Applications Requirements Identify Services & Tools useful for EELA-2 Applications REQUIREMENTS IDENTIFICATION SUPPORT GRID SERVICES DEVELOPMENT NA3 & JRA1 Synergies

4. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 4 EELA-2 Services In the next slides we will see for each EELA-2 Service: A brief introduction Main features Links to installation packages, web/wiki site, documentations, etc. RoadMap All EELA-2 Services will soon be submitted under the evaluation of the EGEE RESPECT programme

5. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 5 GENIUS Gilda Team & Nice S.R.L.

6. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 6 GENIUS GRID portal It can be accessed from everywhere and by “everything” (desktop, laptop, PDA, cell phone). The same user interface to several back-ends. All available grid services incorporated in a logic way, just “one mouse click away”. Layout easily understandable and user friendly. Secure at all levels: –secure for web transactions –secure for user credentials –secure for user authentication –secure at VO level

7. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 7 Main advantages Thanks to the GENIUS and EnginFrame (developed by the Italian Company NICE srl) user can Interact with files on the UI Send jobs to the Grid Manage the data belonging to the given Virtual Organization. Moreover: No needs of a particular Operating System and middle-ware running on the client side

8. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 8 The Genius Home Page (https://glite-tutor.ct.infn.it)

9. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 9 GFAL Java API Diego Scardaci INFN – Catania

10. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 10 GFAL Java API Grid File Acces Library (GFAL): Client POSIX (like) I/O library for directly data access. Aim: Provide GFAL API to Java Developers. Why Java? Reduce developing time; More Java developers are “available”; More applications are “pure Java” applications.

11. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 11 GFAL Java API Classes GFalFile: implements methods to manage remote files (stored on a SE). GFalDirectory: implements methods to manage SE remote directory. GFalUtilities: provides some important utility methods.

12. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 12 References GFAL C API: http://grid-deployment.web.cern.ch/grid-deployment/gis/GFAL/GFALindex.html GFAL Java API wiki page: https://grid.ct.infn.it/twiki/bin/view/GILDA/APIGFAL GFal Java API Javadoc: https://grid.ct.infn.it/twiki/GFAL/

13. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 13 A GRID based platform to host multiple repositories for digital content A. Calanducci, J.M. González, R. Ramos, M. Rubio, D.Tcaci EELA-II Kick-Off Meeting 20th-23rd April 2008 – Trujillo (Spain)

14. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 14 A platform to easily build digital repositories on a grid infrastructure –Digital Repository: annotated digitalized data offered in a structured manner to users Features: –Host multiple repository of arbitrary structure (defined by repository providers) –Totally grid (gLite) based:  Authentication/Authorization based on GSI/VOMS  Digitalized data files saved on any SRM Storage Element  Annotations stored on Metadata Service (AMGA) or RDBMS  Specific algorithm to process data run through the grid WMS –Easy-to-use: AJAX web front-end offering a-la-iTunes browsing –a DRI APIs (Java based) available to easily deploy new repositories What is gLibrary/DRI ?

15. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 15 gLibrary/DRI web interface

16. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 16 One repository must provide A description of its navigational structures (trees, filters) and viewers A description of its data model A storage engine (for data model persistence) The DRI API specification describes HOW this is provided In practise, three Java modules should implement the following interfaces: DRIUIInterface for describing trees, filters and viewers DRIStorageInterface for storing and retrieving data DRINodeInterface for defining repository data model One repository can Make its own implementation of the specification Use the default one provided gLibrary/DRI API

17. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 17 Mammograms rep with viewer

18. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 18 Web interface Servlets, php, java bridge Applets For user authentication with their VO certificate For viewers implementation Java Introspection XML AMGA, gLite Java APIs 18 Technologies involved

19. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 19 Engine deployed and working, API and default implementation (save metadata on AMGA) working –MGPlus repository implemented on DRI –Generic Uploader ready Current work: − Interaction with LFC/SE improvements − Multi-node selection for loading and viewing To Do: –Deployment system for new repositories –Generic viewer for repository that doesn’t provide a specific one –Run grid jobs with selected algorithms with data coming from repositories –Secure Storage/GSAF Integration –Service Orientation for the DRI Engine (Web Service) –Documentation, packaging, installation systems to be RESPECT compliant Current status

20. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 20 gLibrary/DRI presentation at the EGEE User Forum III in Clermont-Ferrand –https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-UF3.pdf https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-UF3.pdf Paper submitted to the 21st International Symposium on Computer Based Medical Systems (CBMS 2008) –https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-CBMS08.pdf https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-CBMS08.pdf Chapter submitted for the “Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine and Healthcare” –https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-chapter.dochttps://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-chapter.doc Poster presented at the User Forum III –https://glibrary.ct.infn.it/glibrary/downloads/DRI/poster_glibraryDRI%20final.pdfhttps://glibrary.ct.infn.it/glibrary/downloads/DRI/poster_glibraryDRI%20final.pdf Developer’s web server: –https://dri-dev.ceta-ciemat.es/index.phphttps://dri-dev.ceta-ciemat.es/index.php Wiki (internals only) –http://www.ceta-ciemat.es/projects/CETA-DRI/http://www.ceta-ciemat.es/projects/CETA-DRI/ References

21. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 21 Grid2Win Porting of gLite middleware to Windows Platform Dario Russo, Fabio Scibilia, Elisa Ingrà INFN –Catania

22. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 22 The Grid2Win project aims to create a gLite User Interface (UI) and a gLite Computing Element (CE + WNs) running on Microsoft Windows. new users come in to the grid ! windows applications can run on the grid ! Grid2Win Cygwin emulates a POSIX environment on Windows; CE based on PBS and Microsoft CCS

23. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 23 We have ported: Grid Security Infrastructure: Credentials management with VOMS extensions commands (voms-proxy-*) Workload Management: Job submission and output retrieving commands (edg-job-*) File Transfer: Secure transfer with credential management GSIFTP (globus-url-copy) RFIO commands (rf*) Data Management: File Catalogue Browsing commands (lfc-*) and data movement & replication (lcg-*) Information System: Command lcg-infosites and existing LDAP Explorer integrated in our GUI. Grid2Win – Main Features

24. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 24 Links You can download the last versione of Grid2Win from the project homepage : http://grid2win.gilda-forge.ct.infn.it/new/index.php Installation and configuration instructions (for VO different than GILDA) can be found here: https://grid.ct.infn.it/twiki/bin/view/GILDA/Grid2WinGUI The online Documentation: http://grid2win.gilda-forge.ct.infn.it/onlineDocumentation/index.php You can read more about the project here: http://grid2win.gilda-forge.ct.infn.it/Grid2Win_ETNGrid2007_final.pdf

25. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 25 T-GSAF Transactional Grid Storage Access Framework Salvatore Scifo Consorzio Cometa – Catania Salvo Parisi IR&T Engineering s.r.l. - Catania

26. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 26 gLite C/C++ API weakness LGC Utils is the only super interface –Doesn’t include GsiFTP –Doesn’t include GFAL –Misses Metadata integration (AMGA) –messy overlap among different semantic APIs  GFAL is only a POSIX-like file manage interface, it should not wrap LFC API than just RFIO would be enough LGC Utils is not a framework LGC Utils is not GOOD for Data Grid Application

27. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 27 GSAF -Data Grid Application solution GSAF is an Object Oriented Framework –Java 100% - Design Pattern based –Clear design : object modeling / functional modeling –Solves the fragmentation of DMS APIs –Solves the natural (due to OGSA) inconsistence of services –Provides ACID Transactional Model (coming soon) Road Map –RFIO java native implementation –SRM encapsulation –Grelc integration –Secure Storage integration New –LFC API –CNS API AMGA is supported only one super interface for Data Grid Application modular and extensible

28. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 28 GSAF as Design Help We have common features, we have common problems  we need a Design Pattern Built on top of the Grid Metadata Service and Grid Data Service –collects and implements functionalities shared among applications according to “write once use anywhere” principle –reduces the knowledge gap  hiding the complexity and the fragmentation of the several underlying APIs  exposing a unified interface more near to the developer mind (design patterns) rather than the Grid stuff details (API syntaxes) –acts as a black box providing  classes and related methods for applications located above  interfaces to extend the implemented capabilities

29. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 29 1° Use Case: DMS Web Interface

30. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 30 Transactional GSAF Transaction Manager –Provides developers a Transaction Pattern  autocommit(), execute(), commit(), rollback()  performs several data manipulation in Atomic Mode –Hold Runtime Exceptions to limit Data Inconsistence –System Crash Recovery to avoid Data Loss Challenge –SOA of Grid middleware makes transactions impossible at server side –GSAF works at client side  Not full access to resources and services (locks)  Not cache mechanism to save temporary data (memory)  Not backup system to store previous data status (restore)  Not full access control to distributed resources (centralization)  …..

31. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 31 State of the Art GSAF –Framework (v1.01) released –Candidate at RESPECT program –Adopters  ADAT Project (Archdiocese of Catania): Cultural Heritage Digital Archive (antique manuscripts)  BM Portal project (Bio-Lab, DIST University of Genoa ) :Service platform to deploy and delivery bio medical and bio informatic applications  Aiuri project (COPPE/UFRJ - BRAZIL)‏:aims to implement a Grid Oriented platform to support data and text mining applications uses GS. –References  http://gilda-forge.ct.infn.it/projects/gsaf/ http://gilda-forge.ct.infn.it/projects/gsaf/  https://grid.ct.infn.it/twiki/bin/view/PI2S2/GSAF https://grid.ct.infn.it/twiki/bin/view/PI2S2/GSAF  http://www.ieeexplore.ieee.org/xpl/freeabs_all.jsp?isnumber=4407105&arnu mber=4407174&count=98&index=68 http://www.ieeexplore.ieee.org/xpl/freeabs_all.jsp?isnumber=4407105&arnu mber=4407174&count=98&index=68 T-GSAF –Analysis and Design (progress) –Development, testing, production (roadmap)

32. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 32 A Secure Storage Service for the gLite Middleware Diego Scardaci – INFN Catania Giordano Scuderi – UNICO S.R.L.

33. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 33 Insider Abuse: the problem A grid user could store sensitive data in a Storage Elements managed by external organizations. Storage Elements Administrators could access data (but the data are sensitive!). For this reason data MUST be stored in an encrypted format. Data Encryption/Decryption MUST be performed inside user secure environment (for example inside the user’s organization).

34. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 34 SE Key Repository SE USER (VIRTUAL) ORGANIZATION Key File Encryption /Decryption Encrypted File Insider Abuse: the solution SECURE ENVIRONMENT

35. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 35 A Secure Storage service for the gLite Middleware Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way. The service is composed by the following components: Command Line Applications: commands integrated in the gLite User Interface to encrypt/upload and decrypt/ download files. Application Program Interface: allows the developer to write programs able to manage confidential data. Keystore: a new grid element used to store and retrieve the users’ keys.

36. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 36 lcg-scr: Encryption and Storage

37. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 37 lcg-scp: Retrieval and Decryption

38. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 38 The Keystore The Keystore is a new grid element used to store and retrieve the users’ key in a secure way. –is identified by an host X.509 digital certificate; –all its Grid transactions are mutually authenticated and encrypted as required by the GSI model; –should be placed in a trusted domain and should be appropriately protected by undesired connections; –is a black box with a single interface towards the external world. This interface accepts only GSI authenticated connections; –the client request is processed only if the client is a member of a enabled users list and/or it belongs to an enabled Virtual Organization; –if the client want to retrieve a key, the keystore checks if the request is coming from an authorized user inserted on the ACL associated to the request key.

39. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 39 Storage Accounting for Grid Environments Fabio Scibilia Consorzio COMETA Catania, Italy

40. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 40 SAGE: Accounting of Storage Resources Accounting in gLite: –CPU: DGAS makes accounting of CPU cycles consumed by jobs. –Storage: Currently no system exists. SAGE: Storage Accounting for Grid Environments –Measures disk space used by files through a scandisk and by intercepting accesses to files –Will be integrated with the database of DGAS (HLR). –Allow users to create reports on the usage of disk space Measure of unit: –Considers both file life time and its space in bytes –Consumption is defined as the integral of the file size (bytes) in time (seconds). http://sage.gilda-forge.ct.infn.it/

41. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 41 Features Interacts with the Mass Storage System (MSS) to resolve local files into grid files. The local MSS (DPM by now) is unaware of being monitored Can be extended to more with more MSS by adding some simple C++ classes to the library It can scale to many disk servers in the same pool It uses MySql as DBMS to store accounting data Accounting is made per file, per user, per VO, per day.

42. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 42 Example of web reporting in SAGE VO admin report generation  Consider this storage  Consider these VO (P.S.: you must be a VO-admin to ask for this) In this time range  Tell me user subjects All the whole site Detailing on these VO Billing of users (one by one) Disk usage in MB*hours

43. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 43 Links SAGE on GILDA forge - http://sage.gilda-forge.ct.infn.it/http://sage.gilda-forge.ct.infn.it/ SAGE on EELA-2 Wiki: - https://grid.ct.infn.it/twiki/bin/view/EELA2/StorageAccounting

44. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 44 WatchDog Monitoring and controlling job execution on the WorkerNode Riccardo Bruno INFN – Catania

45. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 45 Why do we need a WatchDog ? Why –Expecially long term jobs require monitored and controlled during their execution. How –Perform job control and monitoring using grid services in the less invasive way. Observations –Almost all jobs submitted on the grid are piloted by shell scripts  Shell scripting allow to get precious info in case of faults  Shell scripting can pilot more complex batch execution –Both AMGA and SE+LFC can be used as the simplest IS on the grid.  lfc-* and lcg-* tools already available for file creation and retrieve  The latency of CLI tools for the storage is very low compared to long term jobs Requirements: –Monitor job execution watching snapshot of files produced by the job execution  File snapshot will be reported on LFC+SE or AMGA servers –It would be useful to configure the monitoring tool accordingly to the user needs  Few shell environment variables can be used to configure the watchdog tool –Control the job execution accessing directly on the WN  It is possible to send commands on the WN

46. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 46 How does the WatchDog work ? The Watchdog is a shell script to be included in the JDL main script. –Some watchdog features:  It starts in background before to run the long term job  The watchdog runs as long as the main job  The main script can control, stop and wait until the watchdog has finished  Easily and highly configurable and customizable  The watchdog does not compromise the CPU power of the WN Amga LFC+SE Commands or Scripts JDL WN File snapshots WN UI

47. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 47 Links WatchDog wiki page –https://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog

48. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 48 EELA-2 Apps. & JRA1 services (a proposal) (1/2)

49. www.eu-eela.org Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 49 EELA-2 Apps. & JRA1 services (a proposal) (2/2)