1 P.Kunszt C EU DataGrid Data Management Workpackage : WP2 Status and Plans Peter Z Kunszt IT/DB

2 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

3 P.Kunszt C DataGrid workpackages WP1: Workload Management WP2: Grid Data Management WP3: Grid Monitoring Services WP4: Fabric management WP5: Mass Storage Management WP6: Integration Testbed – Production quality International Infrastructure WP7: Network Services WP8: High-Energy Physics Applications WP9: Earth Observation Science Applications WP10: Biology Science Applications WP11: Information Dissemination and Exploitation WP12: Project Management

4 P.Kunszt C Mandate: Data Management The goal of this work package is to specify, develop, integrate and test tools and middleware infrastructure to coherently manage and share petabyte-scale information volumes in high-throughput production-quality grid environments. The work package will develop a general-purpose information sharing solution with unprecedented automation, ease of use, scalability, uniformity, transparency and heterogeneity.

5 P.Kunszt C Mandate: Data Management It will allow to securely access massive amounts of data in a universal global name space, to move and replicate data at high speed from one geographical site to another, and to manage synchronisation of remote replicas. Novel software for automated wide-area data caching and distribution will act according to dynamic usage patterns.

6 P.Kunszt C Data Management Tasks Data Transfer Efficient, secure and reliable transfer of data between sites Data Replication Replicate data consistently across sites Data Access Optimization Optimize data access using replication and remote open Data Access Control Authentication, ownership, access rights on data Metadata Storage Grid-wide persistent metadata store for all kinds of Grid information

7 P.Kunszt C Current Constituency CERN Peter Kunszt, Heinz Stockinger, Leanne Guy, Diana Bosio, Akos Frohner, Wolfgang Hoschek, Kurt Stockinger INFN Flavia Donno, Andrea Domenici, Livio Salconi, Giuseppe Andronico, Federico Di Carlo, Marco Serra PPARC Gavin McCance, William Bell, David Cameron, Paul Millar Trento Cultural Institute Ruben Carvajal Schiaffino, Luciano Serafini, Floriano Zini U.Helsinki/CSC Mika Silander, Joni Hahkala, Ville Nenonen, Niklas Karlsson KDC Stockholm Olle Mulmo, Gian Luca Volpato

8 P.Kunszt C Additional Resources, collaborators LCG:Erwin Laure, Itzhak Ben-Akiva, James Casey PPDG/Condor: Andy Hanushevsky, Aleksandr Konstantinov, Alan Roy Globus/ISI: Ann Chervenak, Bob Schwarzkopf

9 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

10 P.Kunszt C Deliverables DeliverableScheduledDelivered D2.1: tools surveyApril 2001December 2001 D2.2: architecture designJuly 2001September 2001 D2.3: components and documentation for 1 st project release September 2001 In time D2.4: components and documentation for 2 nd project release September 2002 In progress D2.5: components and documentation for 3 rd project release September 2003 Not started

11 P.Kunszt C Communications Quarterly reports on WP2 Regular meetings and workshops –2 DataGrid conferences per year –2 WP2 dedicated workshops per quarter –Weekly management meetings –Weekly phone conferences depending on task 8 dedicated WP2 mailinglists Long list of publications in all of our areas – see www. 26 in total, 12 in journals/proceedings.

12 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

13 P.Kunszt C Collective Services Information & Monitoring Replica Manager Grid Scheduler Replica Optimization Replica Catalog Interface Grid Application Layer Job Management Local Application Local Database Fabric services Configuration Management Configuration Management Node Installation & Management Node Installation & Management Monitoring and Fault Tolerance Monitoring and Fault Tolerance Resource Management Fabric Storage Management Fabric Storage Management Grid Fabric Local Computing Grid Data Management Metadata Management Object to File Mapper Underlying Grid Services Computing Element Services Authorisation, Authentication and Accounting Replica Catalog Storage Element Services Metadata Service Service Index

14 P.Kunszt C File Management Site A Storage Element AStorage Element B Site B File B File AFile X File YFile B File AFile C File D Replica Catalog: Map Logical to Physical files File Transfer Replica Manager: ‘atomic’ replication operation single client interface orchestrator Pre- Post-processing: Prepare files for transfer Validate files after transfer Replica Selection: Get ‘best’ file Replication Automation: Data Source subscription Load balancing: Replicate based on usage Metadata: LFN metadata Transaction information Access patterns

15 P.Kunszt C Architecture Replica Location Index Site Replica Manager Storage Element Computing Element Optimiser Resource Broker User Interface Pre-/Post- processing Core API Optimisation API Processing API Local Replica Catalogue Replica Location Index Replica Metadata Catalogue Replica Location Index Site Replica Manager Storage Element Computing Element Optimiser Pre-/Post- processing Local Replica Catalogue

16 P.Kunszt C Current Components File Transfer: Use GridFTP – deployed –Close collaboration with Globus –NetLogger (Brian Tierney and John Bresnahan) Replication: GDMP – deployed –Wrapper around Globus ReplicaCatalog –All functionality in one integrated package –Using Globus 2 –Uses GridFTP for transferring file Replication: edg-replica-manager – deployed Replication: Replica Location Service Giggle – in testing –Distributed Replica Catalog Replication: Replica Manager Reptor – in testing Optimization: Replica Selection OptorSim – in simulation Metadata Storage: SQL Database Service Spitfire – deployed –Servlets on HTTP(S) with XML (XSQL) –GSI enabled access + extensions GSI interface to CASTOR – delivered

17 P.Kunszt C Current Status GDMP version –Support for multiple VOs –New security mechanism for server and clients –uses Globus 2.0 beta 21 on EDG testbed –Linux RedHat 6.2 (6.1.1) partly already RedHat 7.2 (some manual changes required) –GDMP is part of VDT (Virtual Data Toolkit) –Alain Roy (Condor team) does support for US sites Replica Manager –Wrapper around existing Globus replica manager edg- replica-manager –Core API as defined in Replica Manager Design Document is implemented Reptor Java Servlet, alpha release in internal testing Replication

18 P.Kunszt C Current Status Replica Catalog –GUI and command line tool for existing RC Available to the production testbed –Introduced a security patch for RC –Collaboration with Globus and PPDG on new, distributed Replica Catalog Framework (Giggle) joint design with Globus, G:coding, EDG:testing Internal alpha release, integrated with Reptor Working prototype of Simulator Replication cont. Optimization

19 P.Kunszt C File mappings LogicalPhysical PFN 1 PFN 2 PFN 3 PFN n LFN 1 LFN 2 LFN 3 LFN n LFN 0 / Grid Unique ID RLSRepMeC A File

20 P.Kunszt C Replica location problem Replica Location Service: –maintains information on the physical location of files –maintains mapping between the logical identifier of data and all its physical instances –provides access to this information Given a unique logical identifier for some given data, determine the physical location of one or more physical instances of this data

21 P.Kunszt C RLS Requirements  Versioning & read only data –distinct versions of files can be uniquely identified –data published to the community are immutable  Size –scale to hundreds of replica sites, 50 x 10 8 LFNs, 500 x 10 8 PFNs  Performance –200 updates/second, average response time < 10ms  Security –knowledge and existence of private data must be protected –storage system protects integrity of data content  Consistency –view of all available PFNs not necessarily consistent  Reliability –no single point of failure, –local and global state decoupled, failure of remote component does not hinder access to local component

22 P.Kunszt C Giggle Framework Giggle: A Framework for Constructing Scalable Replica Location Services – Joint collaboration between WP2 and Globus – Paper submitted to SC2002 LRCs  Independent local state maintained in Local Replica Catalogues : LRCs RLIs  Unreliable collective state maintained in Replica Location Indices : RLIs  Soft state maintenance of RLI state –relaxed consistency in the RLI, full state information in LRC  Compression of soft states –compress LFN information based on knowledge of logical collections  Membership and partitioning information maintenance –RLS components change over time : failure, new components added –Service discovery and system policies

23 P.Kunszt C Local Replica Catalogue (LRC) Maintains replica information at a single site –Complete locally consistent record –Queries across multiple sites not supported Maintains mappings between LFNs and PFNs on associated storage systems –Coordinates its contents with those of the storage system Responds to the following queries : –Given an LFN, find the set of PFNS associated with that LFN –Given a PFN, find the set of LFNS associated with that PFN Supports authentication and authorisation when processing remote requests Periodically sends information about its state to the RLIs

24 P.Kunszt C Replica Location Index (RLI) Index structure needed to support queries across multiple sites One or more RLIs to map LFNs to LRCs –Structure w.r.t LRCs can be freely defined –redundancy, performance, scalability Geographical partitioning – all PFNs of a set of LRCs are indexed Namespace partitioning 1 – for load balancing purposes Namespace partitioning 2 – only LFNs adhering to a specified pattern are indexed for all LRCs –possibly not good for load balancing Many identical RLIs may be set up for load balancing

25 P.Kunszt C RLS Architecture (1) A 2 level RLS layout: The RLIs contain pointers to LRCs only. LRC RLI LRC Multiply indexed LRC for higher availability RLI indexing over the full namespace (all LRCs are indexed) RLI indexing over a subset of LRCs LRC indexed by only one RLI

26 P.Kunszt C RLS Architecture (2) A hierarchical RLS topology: The RLIs point to LRCs and RLIs RLI LRC RLI LRC Multiply indexed LRC for higher availability RLI indexing over the full namespace (all LRCs are indexed) RLI indexing over a subset of LRCs LRC indexed by only one RLI

27 P.Kunszt C Metadata Management and Security Project Spitfire 'Simple' Grid Persistency –Grid Metadata –Application Metadata –Unified Grid enabled front end to relational databases. Metadata Replication and Consistency Publish information on the metadata service Secure Grid Services Grid authentication, authorization and access control mechanisms enabled in Spitfire Modular design, reusable by other Grid Services

28 P.Kunszt C Spitfire Architecture OracleDB2PostGresMySQL Atomic RDBMS is always consistent No local replication of data Role-based authorization XSQL Servlet as one access mode for ‘simple’ web access Web/Grid Services Paradigm –SOAP interfaces –JDBC interface to RDBMS Pluggability and extensibility OracleLayerDB2LayerPGLayerMyLayer Local Spitfire Layer Connecting Layer Global Spitfire Layer SOAP

29 P.Kunszt C Security Mechanism Servlet Container SSLServletSocketFactory TrustManager Security Servlet Does user specify role? Map role to connection id Authorization Module HTTP + SSL Request + client certificate Yes Role Trusted CAs Is certificate signed by a trusted CA? No Has certificate been revoked? Revoked Certs repository Find default No Role repository Role ok? Connection mappings Translator Servlet RDBMS Request and connection ID Connection Pool

30 P.Kunszt C Status Integrated XSQL-Spitfire including security mechanisms XSQL version of a Replica MetaData Catalog –Schema is given –Roles are fixed –Queries are predefined –No SOAP, just XML over http Installation decoupled from Tomcat & MySQL Work on SOAP interface using axis Security mechanisms planned Spitfire & Security

31 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

32 P.Kunszt C WP2 Replication Services Replica Manager Replica Metadata Replica Location File Transfer Optimization Transaction Consistency Preprocessing Postprocessing Subscription Client Reptor Giggle RepMeC Optor GDMP

33 P.Kunszt C Plans Service-type Replica Manager integrated with Giggle and Replica Metadata Catalog Optimization module integrated Automated replication module designed and prototyped Replication & Optimization

34 P.Kunszt C Plans All functionality available Security fully integrated Installation works with more than just Tomcat and MySQL Test-suite completely implemented Working Replication Metadata Catalog using this Beta, integration with Replica Manager Spitfire & Security

35 P.Kunszt C Summary of Plans GDMP v3.0 will be the LAST release of GDMP as we know it. GDMP in the future will rely on the Replica Manager and provide the subscription based mirroring functionality. It will be implemented as a web service. The Replica Catalog will be replaced with the Giggle framework, jointly developed with Globus. The Replica Manager will take over most of GDMPs current functionality and will be a web service. We provide client APIs. All user interaction should go through the RM. Spitfire will have both an XML over http access method for static queries and a SOAP interface. Security infrastructure for all services in place as presented.

36 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

37 P.Kunszt C Issues Getting everybody started, coordination with partners – one very late deliverable Technology challenges – evolving field, keeping on the top needs a lot of training effort of everybody Coordination within WP2 and US projects – not enough funds for CERN Very late arrival of and changes in manpower – additional training and supervision A lot of young members are working / have worked on their PhD Different agenda of user community delays development: continuous requests for support and enhancements of components that we want to phase out (GDMP) takes out development manpower, documentation suffers most.

38 P.Kunszt C Outlook Very ambitious programme, can we make it? Current workforce is very well motivated and has a very high level of interaction – see our mailinglist archives A lots of enhancements in inter-task coordination was done, further effort is necessary – workshops THANK YOU FOR YOUR ATTENTION