1. Grid DataBase Management with the GRelC DAS Sandro Fiore, Ph.D sandro.fiore@unile.it Salvatore Vadacca salvatore.vadacca@unile.it SPACI Consortium and University of Salento (Lecce), Italy Grid Tutorial per l’Università di Palermo, 11/12/2007

2. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 2 Outline Grid Data Access Service GRelC Project GRelC DAS –Architecture –Features –Queries –SDK –Experimental Results –Porting on gLite –XGRelC GUI –GRelC Portal –Deployment –On Line User Tutorial (GILDA) Conclusions

3. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 3 In the last decade a lot of efforts were concentrated on grid storage systems (coarse grained approach) Data Grids should also provide a low level framework for grid- database access and management (Grid DAS, fine grained approach) A Grid DAS is not a new DBMS or it does not introduce a new query language It interacts with legacy systems/databases (SQL) It provides more complex and efficient “queries” in grid It must be interoperable with current standards/middleware (gLite, Globus, …) Main requirements: security, transparency, interoperability, efficiency, robustness, etc. Grid Data Access Service

4. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 4 Introducing the GRelC Project Grid Relational Catalog is a project which aims at designing and developing a set of efficient, secure and transparent Data Grid Services (Starting date, Jan 2001). GRelC Data Access Service aims at providing a large set of functionalities to access both relational and non relational DataBases in a grid environment. DB XML

5. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 5 GRelC Project: a bit of history

6. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 6 GRelC DAS Architecture (in the large)

7. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 7 GRelC DAS: Main Features Entirely based on C programming language Multithreaded web service It exposes the web service interface GSI enabled and WS-I compliant Mutual authentication based on GSI (X.509v3 digital certificates) GRelC DAS Authorization based on ACL for local management VOMS Support, for global management Information System Support (BDII compliant) Wide set of data access control policyies Full GSI support: data encryption, data integrity, protection against replay attacks and detection of out of sequence packets

8. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 8 GRelC DAS: Main Features XML data validation for recordset SingleQuery, MultiQuery and MultiSingleQuery Support Support for synchronous and asynchronous queries Dinamic binding to heterogeneous DBMSs Two levels logging (users, connections, queries, etc.) GSI enabled remote administration tools and remote log Compression, chunking, prefetching and streaming to enhance performance on a WAN Wide SDK for developers (both for C, C++ and Java) No dependencies concerning other middleware (only GSI) Portal GRelC

9. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 9 GRelC DAS Architecture (in the small) GRelC Data Access Service

10. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 10 Standard Database Access Interface Features: Standard access to data sources Types uniformity Error uniformity Plug-in architecture based on dynamic libraries PostgreSQL driver MySQL driver Database Access Library (Grid-DAS back end) Grid Database Access Service (front end) Other Applications PostgreSQL MySQL UnixODBC driver UnixODBC SQLite driver SQLite Dinamic binding to: PostgreSQL MySQL SQLite IBM/DB2, Oracle9.i, MS-SQL Server, UnixODBC, Textual DBs, etc.

11. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 11 GRelC DAS Queries Latest release of GRelC DAS supports the following queries: Single Query Memory (+ chunk management) Single Query File (+ chunk management) Single Query File + ZIP (+ chunk management) Single Query Prefetch (parallel chunk donwload/processing) Single Query Stream (resultset streaming) Asynchronous Query Web Single Query XHTML (+ chunk management / paging) –CSS v2.0, XHTML v1.0 Strict Results displayed in the following formats: Tabular XML HTML RAW

12. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 12 Single Query File Approach (Zip) DBMS Client SQS query Data Delivery SQL Recordset GrelC Recordset in XMLformat GrelC Recordset GrelCLoad Recordset GrelCRecordset APIs GRelC Data Access Recordset XML File Recordset XML File This kind of query is suitable to retrieve medium/large resultsets

13. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 13 Single Query File + chunk (Zip) DBMS Client SQS query Data Delivery SQL Recordset GrelC Recordset in XMLformat GrelC Recordset GrelCLoad Recordset GrelCRecordset APIs GRelC Data Access Recordset XML file Recordset XML file Recordset XML file Chunk 1 Chunk 2 Chunk 3 Recordset XML file Recordset XML file Recordset XML file Chunk 1 Chunk 2 Chunk 3 Chunk 1 Chunk 2 Chunk 3 Chunk 1 Chunk 2 Chunk 3 This kind of query is suitable to retrieve medium/large resultsets

14. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 14 Single Query Stream DBMS Client SQS submission SQL Recordset GRelC-Data-Access Result submision This kind of query is suitable to retrieve VERY LARGE resultsets

15. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 15 Single Query HTML DBMS Client SQS submission SQL Recordset Http connection GRelC-Data-Access URI Result

16. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 16 Web Page GACL DNs Single Query HTML (Security Issues) DBMS Client SQS submission SQL Recordset HTTPS connection using X.509 Certificates GRelC-Data-Access URI Result

17. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 17 Asynchronous Query Asynchronous queries - Batch mode - Users can define a lifetime for results availability on the GRelC DAS - decoupling client/server (e.g. WN gLite) - New clients (submission, status, abort…) - Additional thread to manage requests - Preliminary internal tests were ok - Added within the current release v2.3.0

18. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 18 Asynchronous Query DBMS Client SQS query ID Query SQL Recordset GrelC Recordset in XMLformat GrelC Recordset GrelCLoad Recordset GrelCRecordset APIs GRelC Data Access Recordset XML File Recordset XML File ID Query Get File Data Delivery 1 – Asynchronous Query Submission 2 – Request Dispatching 3 – Data Delivery 4 – Data Manipulation

19. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 19 GRelC Data Access: Clients

20. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 20 SDK for Developers Grid Enabled Stand alone GUI (XGRelC) Grid Enabled Web Apps (GRelC Portal) High Performance Grid Services & Clients (CLI) GRelC DAS

21. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 21 The GRelC Library: APIs Classification Database access and query services – bind – unbind – query submission Remote manipulation services – get_value – get_current_tuple Resultset store and retrieving services – store_result_disk – fetch_stored_recordset User management services – add_user – remove_user – set_user_policy Enterprise Grid management services – add_host – add_dbms Virtual space management services – create_virtual_database – register_database QoS services – relocate_database Wide SDK both for C and C++ developers

22. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 22 GRelC-CppProxy: a C++ Module A C++ module was created in order to allow an easy development of new web services client with this language. This module hides the communication layer with Web Services C++ Module C++ Generic Client SOAP GRelC DAS

23. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 23 class CppProxy { public: int bind (string grelc_db_name); int unbind (); int query_submission (string query);.... int create_database (string grelc_db_name, string identity, string dbms, string host, string istance, int log_type); int create_physical_database_and_register(string grelc_db_name, string dbms, string host, string istance, int log_type); int drop_database (string grelc_db_name); int get_log(int num_lines, string &log); int get_log_database(string grelc_dbname, int num_lines, string &log); int get_host_position_info(HostInfoResponse &response); int get_value (int row, int column, string &value); private: struct soap *soap; struct gsi_plugin_data *data; char* connection; bool connected; bool enable_credential; string dn; }; CppProxy Class Low level details concerning with soap, gsi, conn. status are concealed

24. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 24 XGRelC: A consolle for Grid-DBs Mng Functionalities: User management Web Service registration Host Management Logging DBMS configuration Database creation Import Database Database configuration Query submission Map deployment

25. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 25 XGRelC: A consolle for Grid-DBs Mng Functionalities: User management Web Service registration Host Management Logging DBMS configuration Database creation Import Database Database configuration Query submission Map deployment

26. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 26 International Testbed Lecce (Italy) Bejing (China)

27. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 27 Test Performance (II)

28. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 28 GRelC & gLite

29. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 29 GRelC on gLite: Porting Porting of GRelC on gLite was straighforward Porting on gLite is ok both for client and server side The middleware works fine both on LCG-2-7-0 and current gLite 3.x middleware GRelC DAS runs also on several platforms: –Linux –MAC OS X –FreeBSD Both IA64 and IA32 platforms are supported (we currently installed on SPACI-LECCE-IA64 (EGEE SA1 partner) the GRelC DAS)

30. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 30 Straighforward integration within the EGEE farm model GRelC DAS provides fine grained data mng service This service can be used both as farm service and as VO service depending on the context, the database policies/constraints Extended EGEE Farm Model GRelC on gLite: A New Service

31. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 31 GRelC on gLite: VOMS We provide global authorization by means of VOMS Extensions High level of scalability concerning DAPs related to VOs Double level authorization framework: both local and global policies management can be provided (mixed mode) gilda grelc das host1 host2host3 grid-db1 grid-db2 Root level (VO) GRelC level DAS Level Host Level Grid-DB Level Roles Mandatory Groups insert delete update Fine Grained Coarse Grained

32. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 32 Two-level authorization Global authorization (through VOMS extensions) Local authorization (by means of the local GRelC DAS authorization framework) The two masks obtained from global and local authorization are combined to infer the final User Privileges Mask (UPM) 3 scenarios –global mode, coarse grained approach –local mode, fine grained approach –combined mode

33. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 33 Combined Mode - An Example

34. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 34 GRelC on gLite: BDII GLUE schema extension providing information about VOs and Databases (we plan to interact with OGF GLUE-WG) Local admin can set up the Information Provider Level parameter Min: 0 to publish just basic info (only the contact string) Max: 7 for all info (contact string, VOs, DBs, tables, fields, etc.) Information System Extensions Database specific Information

35. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 35 GRelC on gLite: Porting on SLC4.x Porting on SLC4.x is an on going activity Preliminary results are very good Porting will be completed before EGEE Conference in Budapest A release based on SLC4.x will be available on the GRelC website in December Current test are connected both with IA32 and IA64 (Itanium2 processors) platforms This activity is part of the SPACI-LECCE-IA64 SA1 activity within the EGEE Project

36. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 36 GRelC Portal Functionalities:  Login  GRelC DAS Registration  Host Management  Instance Management  User Management  Query submission  Deployment Map Features:  Seamless and ubiquitous access to GRelC DAS enabled resources  No additional software installation / configuration is required  Complete and user-friendly Grid Data Portal Interface (It entirely replaces CLI)  https://grid.ct.infn.it/twiki/bin/view/GILDA/GRelCPortal  https://grid.ct.infn.it/twiki/bin/view/GILDA/GRelCPortalAdvanced

37. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 37 GRelC Portal: Some Snapshots

38. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 38 GRelC WebSite Main sections: GRelC Portal Download (rpms available) News Publications Events Deployment Documentation Components ….. GRelC Website URL: http://grelc.unile.it/http://grelc.unile.it/ Mailing List mail: grelc-user@sara.unile.itgrelc-user@sara.unile.it

39. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 39 INFN GRID Deployment SPACI Lecce DAS Server DAS Client INAF Trieste INFN Padova INFN Bari INFN Catania Involved Sites INAF Trieste (IA32) INFN Bari (IA32) INFN Catania (IA32) INFN Padova (IA32) SPACI LECCE (IA32, IA64) Testing Activities Sequential tests Concurrent tests Bugs report Bug Fixing Optimization SPACI-Lecce

40. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 40 GRelC Data Access Data Sources (DB) gandalf.unile.it Linux x86 sara.unile.it Mac OS X sigma2.unile.it Linux IA64 gridsurfer.unile.it FreeBSD galileo.hpcc.unical.it Linux IA64 sepac00.projects.cscs.ch Linux x86 spacina.na.infn.it Linux IA64 SEPAC Grid Deployment

41. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 41 GILDA Deployment

42. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 42 GRelC DAS User Tutorial on GILDA Grid CT Wiki Website Info about: -Log in to the grid -Query Submission For any information about GILDA t-Infrastructure please contact roberto.barbera@ct.infn.it & grid-prod@ct.infn.itroberto.barbera@ct.infn.itgrid-prod@ct.infn.it GRelC DAS Tutorial link: https://grid.ct.infn.it/twiki/bin/view/GILDA/GRelCDataAccessService https://grid.ct.infn.it/twiki/bin/view/GILDA/GRelCDataAccessService User tutorial: GILDA t-Infrastructure Special thanks to the GILDA Staff for their support

43. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 43 Christmas Release Info Tag: GRelC DAS v2.4.0 Release date: 21/12/2007 Support: SCL3.x & SLC4.x Features: New driver for SQLite New GRelC DAS and Grid-DB Log Remote Log Support XML Support for –eXist –Xindice –libxml2 based documents Complete set of CLI for XML New GILDA Tutorial for XML CLI XML Grid Information Provider What’s Next… GRelC DAS 2.4.0

44. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 44 Conclusions GRelC DAS provides support in Grid for a wide range of DBMSs It is currently tested on several grid environments (SPACI, SEPAC, GILDA, INFNGRID) A wide SDK is available for developers XGRelC Graphical User Interface developed in Qt GRelC Portal to ease Grid-DB mng via Web Interface gLite compliant (porting on gLite 3.x and integration with VOMS framework, BDII, etc.) Support for several platforms (IA32 and IA64) Currently the software is candidate at the EGEE Respect Program

45. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 45 For any information Supervisor: Prof. Giovanni Aloisio (giovanni.aloisio@unile.it)giovanni.aloisio@unile.it Project P. I.: Sandro Fiore, Ph. D. (sandro.fiore@unile.it)sandro.fiore@unile.it Team Members: Massimo Cafaro, Ph. D. Alessandro Negro Salvatore Vadacca GRelC WebSite: http://grelc.unile.ithttp://grelc.unile.it Mailing lists: grelc-user@sara.unile.itgrelc-user@sara.unile.it

46. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 46 Global Mode User credentials must be obtained through voms- proxy-init The UPM is inferred from the available VOMS extensions No additional authorization setting is required on the GRelC DAS Easy and fast setup procedure It scales well Feasible for a real production grid environment

47. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 47 Global Mode

48. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 48 Local Mode User credentials must be obtained through grid- proxy-init The UPM is drawn out of the GRelC DAS metadata catalogue No VOMS extensions are added to the user proxy The setup procedure must be carried out on each GRelC DAS Scalability is worse

49. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 49 Local Mode

50. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 50 Combined Mode User credentials must be obtained through voms- proxy-init The UPM is inferred joining information on access policies coming from VOMS extensions and the GRelC DAS metadata catalogue VOMS level (grant or revoke) GRelC DAS level (setting, undefining, unsetting)

51. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 51 Roles and Groups on VOMS (I) /gilda/grelc/das/host1/grid-db1/Role=grelc-db-insert Case A (fine grained)

52. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 52 Roles and Groups on VOMS (II) /gilda/grelc/das/host1/Role=grelc-db-insert Case B (intermediate level)

53. Grid Tutorial, Palermo - Ph.D. Sandro Fiore - GRelC Project - http://grelc.unile.it 53 Roles and Groups on VOMS (III) /gilda/grelc/das/Role=grelc-db-insert Case C (coarse grained)