1 P.Kunszt C5 14.6.2002 EU DataGrid Data Management Workpackage : WP2 Status and Plans Peter Z Kunszt IT/DB

Slides:



Advertisements
Similar presentations
Giggle: A Framework for Constructing Scalable Replica Location Services Ann Chervenak, Ewa Deelman, Ian Foster, Leanne Guy, Wolfgang Hoschekk, Adriana.
Advertisements

The Replica Location Service In wide area computing systems, it is often desirable to create copies (replicas) of data objects. Replication can be used.
1 WP2: Data Management Paul Millar eScience All Hands Meeting September
WP2: Data Management Gavin McCance University of Glasgow November 5, 2001.
WP2: Data Management Gavin McCance University of Glasgow.
EU DataGrid TestBed 2 Component Review Paul Millar (University of Glasgow) (slides based on a presentation by Erwin Laure)
WP2 and GridPP UK Simulation W. H. Bell University of Glasgow EDG – WP2.
Data Management Expert Panel - WP2. WP2 Overview.
Data Management Expert Panel. RLS Globus-EDG Replica Location Service u Joint Design in the form of the Giggle architecture u Reference Implementation.
Andrew McNab - Manchester HEP - 2 May 2002 Testbed and Authorisation EU DataGrid Testbed 1 Job Lifecycle Software releases Authorisation at your site Grid/Web.
DataGrid is a project funded by the European Commission under contract IST WP2 – R2.1 Overview of WP2 middleware as present in EDG 2.1 release.
High Performance Computing Course Notes Grid Computing.
RLS Production Services Maria Girone PPARC-LCG, CERN LCG-POOL and IT-DB Physics Services 10 th GridPP Meeting, CERN, 3 rd June What is the RLS -
DataGrid is a project funded by the European Union 22 September 2003 – n° 1 EDG WP4 Fabric Management: Fabric Monitoring and Fault Tolerance
EU-GRID Work Program Massimo Sgaravatto – INFN Padova Cristina Vistoli – INFN Cnaf as INFN members of the EU-GRID technical team.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
10 April 2003Deploy Grid in Israel Universities1 Deploy Grid testbed in Israel universities Lorne Levinson David Front Weizmann Institute.
DGC Paris Community Authorization Service (CAS) and EDG Presentation by the Globus CAS team & Peter Kunszt, WP2.
DataGrid is a project funded by the European Union CHEP 2003 – March 2003 – Title – n° 1 Grid Data Management in Action Experience in Running and.
Workload Management Workpackage Massimo Sgaravatto INFN Padova.
GGF Toronto Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.
DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.
UMIACS PAWN, LPE, and GRASP data grids Mike Smorul.
The EU DataGrid Architecture The European DataGrid Project Team
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
GridPP9 – 5 February 2004 – Data Management DataGrid is a project funded by the European Union GridPP is funded by PPARC WP2+5: Data and Storage Management.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
Presenter: Dipesh Gautam.  Introduction  Why Data Grid?  High Level View  Design Considerations  Data Grid Services  Topology  Grids and Cloud.
A Metadata Catalog Service for Data Intensive Applications Presented by Chin-Yi Tsai.
DISTRIBUTED COMPUTING
ARGONNE  CHICAGO Ian Foster Discussion Points l Maintaining the right balance between research and development l Maintaining focus vs. accepting broader.
GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.
1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong
DataGrid is a project funded by the European Union CHEP 2003 – March 2003 – Next Generation Data Mgmt... – n° 1 James Casey CERN
ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.
Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.
Grid Workload Management Massimo Sgaravatto INFN Padova.
CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.
WP9 – Earth Observation Applications – n° 1 WP9 report to Plenary ESA, KNMI, IPSL Presented by M. Petitdidier, IPSL DataGrid Plenary Session 5 th Project.
Virtual Data Grid Architecture Ewa Deelman, Ian Foster, Carl Kesselman, Miron Livny.
1 WP2: Data Management Gavin McCance RAL Middleware Workshop 24 February 2003.
Author - Title- Date - n° 1 Partner Logo EU DataGrid, Work Package 5 The Storage Element.
Author - Title- Date - n° 1 Partner Logo WP5 Summary Paris John Gordon WP5 6th March 2002.
Part Four: The LSC DataGrid Part Four: LSC DataGrid A: Data Replication B: What is the LSC DataGrid? C: The LSCDataFind tool.
09/02 ID099-1 September 9, 2002Grid Technology Panel Patrick Dreher Technical Panel Discussion: Progress in Developing a Web Services Data Analysis Grid.
The Replica Location Service The Globus Project™ And The DataGrid Project Copyright (c) 2002 University of Chicago and The University of Southern California.
GridPP Presentation to AstroGrid 13 December 2001 Steve Lloyd Queen Mary University of London.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Replica Management Services in the European DataGrid Project Work Package 2 European DataGrid.
CLRC and the European DataGrid Middleware Information and Monitoring Services The current information service is built on the hierarchical database OpenLDAP.
May http://cern.ch/hep-proj-grid-fabric1 EU DataGrid WP4 Large-Scale Cluster Computing Workshop FNAL, May Olof Bärring, CERN.
GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.
1 P.Kunszt LCGP Data Management on the GRID Peter Z. Kunszt CERN Database Group EU DataGrid – Data Management.
Data Management GridPP and EDG Gavin McCance University of Glasgow May 9, 2002
DGC Paris WP2 Summary of Discussions and Plans Peter Z. Kunszt And the WP2 team.
6/23/2005 R. GARDNER OSG Baseline Services 1 OSG Baseline Services In my talk I’d like to discuss two questions:  What capabilities are we aiming for.
Oracle to MySQL synchronization Gianni Pucciani CERN, University of Pisa.
David Foster LCG Project 12-March-02 Fabric Automation The Challenge of LHC Scale Fabrics LHC Computing Grid Workshop David Foster 12 th March 2002.
Data Management The European DataGrid Project Team
DGC Paris Spitfire A Relational DB Service for the Grid Leanne Guy Peter Z. Kunszt Gavin McCance William Bell European DataGrid Data Management.
1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.
Current Globus Developments Jennifer Schopf, ANL.
Bob Jones – Project Architecture - 1 March n° 1 Project Architecture, Middleware and Delivery Schedule Bob Jones Technical Coordinator, WP12, CERN.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
WP2: Data Management Gavin McCance University of Glasgow.
Gavin McCance University of Glasgow GridPP2 Workshop, UCL
StoRM: a SRM solution for disk based storage systems
The Work Package 2 experience
Patrick Dreher Research Scientist & Associate Director
Presentation transcript:

1 P.Kunszt C EU DataGrid Data Management Workpackage : WP2 Status and Plans Peter Z Kunszt IT/DB

2 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

3 P.Kunszt C DataGrid workpackages WP1: Workload Management WP2: Grid Data Management WP3: Grid Monitoring Services WP4: Fabric management WP5: Mass Storage Management WP6: Integration Testbed – Production quality International Infrastructure WP7: Network Services WP8: High-Energy Physics Applications WP9: Earth Observation Science Applications WP10: Biology Science Applications WP11: Information Dissemination and Exploitation WP12: Project Management

4 P.Kunszt C Mandate: Data Management The goal of this work package is to specify, develop, integrate and test tools and middleware infrastructure to coherently manage and share petabyte-scale information volumes in high-throughput production-quality grid environments. The work package will develop a general-purpose information sharing solution with unprecedented automation, ease of use, scalability, uniformity, transparency and heterogeneity.

5 P.Kunszt C Mandate: Data Management It will allow to securely access massive amounts of data in a universal global name space, to move and replicate data at high speed from one geographical site to another, and to manage synchronisation of remote replicas. Novel software for automated wide-area data caching and distribution will act according to dynamic usage patterns.

6 P.Kunszt C Data Management Tasks Data Transfer Efficient, secure and reliable transfer of data between sites Data Replication Replicate data consistently across sites Data Access Optimization Optimize data access using replication and remote open Data Access Control Authentication, ownership, access rights on data Metadata Storage Grid-wide persistent metadata store for all kinds of Grid information

7 P.Kunszt C Current Constituency CERN Peter Kunszt, Heinz Stockinger, Leanne Guy, Diana Bosio, Akos Frohner, Wolfgang Hoschek, Kurt Stockinger INFN Flavia Donno, Andrea Domenici, Livio Salconi, Giuseppe Andronico, Federico Di Carlo, Marco Serra PPARC Gavin McCance, William Bell, David Cameron, Paul Millar Trento Cultural Institute Ruben Carvajal Schiaffino, Luciano Serafini, Floriano Zini U.Helsinki/CSC Mika Silander, Joni Hahkala, Ville Nenonen, Niklas Karlsson KDC Stockholm Olle Mulmo, Gian Luca Volpato

8 P.Kunszt C Additional Resources, collaborators LCG:Erwin Laure, Itzhak Ben-Akiva, James Casey PPDG/Condor: Andy Hanushevsky, Aleksandr Konstantinov, Alan Roy Globus/ISI: Ann Chervenak, Bob Schwarzkopf

9 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

10 P.Kunszt C Deliverables DeliverableScheduledDelivered D2.1: tools surveyApril 2001December 2001 D2.2: architecture designJuly 2001September 2001 D2.3: components and documentation for 1 st project release September 2001 In time D2.4: components and documentation for 2 nd project release September 2002 In progress D2.5: components and documentation for 3 rd project release September 2003 Not started

11 P.Kunszt C Communications Quarterly reports on WP2 Regular meetings and workshops –2 DataGrid conferences per year –2 WP2 dedicated workshops per quarter –Weekly management meetings –Weekly phone conferences depending on task 8 dedicated WP2 mailinglists Long list of publications in all of our areas – see www. 26 in total, 12 in journals/proceedings.

12 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

13 P.Kunszt C Collective Services Information & Monitoring Replica Manager Grid Scheduler Replica Optimization Replica Catalog Interface Grid Application Layer Job Management Local Application Local Database Fabric services Configuration Management Configuration Management Node Installation & Management Node Installation & Management Monitoring and Fault Tolerance Monitoring and Fault Tolerance Resource Management Fabric Storage Management Fabric Storage Management Grid Fabric Local Computing Grid Data Management Metadata Management Object to File Mapper Underlying Grid Services Computing Element Services Authorisation, Authentication and Accounting Replica Catalog Storage Element Services Metadata Service Service Index

14 P.Kunszt C File Management Site A Storage Element AStorage Element B Site B File B File AFile X File YFile B File AFile C File D Replica Catalog: Map Logical to Physical files File Transfer Replica Manager: ‘atomic’ replication operation single client interface orchestrator Pre- Post-processing: Prepare files for transfer Validate files after transfer Replica Selection: Get ‘best’ file Replication Automation: Data Source subscription Load balancing: Replicate based on usage Metadata: LFN metadata Transaction information Access patterns

15 P.Kunszt C Architecture Replica Location Index Site Replica Manager Storage Element Computing Element Optimiser Resource Broker User Interface Pre-/Post- processing Core API Optimisation API Processing API Local Replica Catalogue Replica Location Index Replica Metadata Catalogue Replica Location Index Site Replica Manager Storage Element Computing Element Optimiser Pre-/Post- processing Local Replica Catalogue

16 P.Kunszt C Current Components File Transfer: Use GridFTP – deployed –Close collaboration with Globus –NetLogger (Brian Tierney and John Bresnahan) Replication: GDMP – deployed –Wrapper around Globus ReplicaCatalog –All functionality in one integrated package –Using Globus 2 –Uses GridFTP for transferring file Replication: edg-replica-manager – deployed Replication: Replica Location Service Giggle – in testing –Distributed Replica Catalog Replication: Replica Manager Reptor – in testing Optimization: Replica Selection OptorSim – in simulation Metadata Storage: SQL Database Service Spitfire – deployed –Servlets on HTTP(S) with XML (XSQL) –GSI enabled access + extensions GSI interface to CASTOR – delivered

17 P.Kunszt C Current Status GDMP version –Support for multiple VOs –New security mechanism for server and clients –uses Globus 2.0 beta 21 on EDG testbed –Linux RedHat 6.2 (6.1.1) partly already RedHat 7.2 (some manual changes required) –GDMP is part of VDT (Virtual Data Toolkit) –Alain Roy (Condor team) does support for US sites Replica Manager –Wrapper around existing Globus replica manager edg- replica-manager –Core API as defined in Replica Manager Design Document is implemented Reptor Java Servlet, alpha release in internal testing Replication

18 P.Kunszt C Current Status Replica Catalog –GUI and command line tool for existing RC Available to the production testbed –Introduced a security patch for RC –Collaboration with Globus and PPDG on new, distributed Replica Catalog Framework (Giggle) joint design with Globus, G:coding, EDG:testing Internal alpha release, integrated with Reptor Working prototype of Simulator Replication cont. Optimization

19 P.Kunszt C File mappings LogicalPhysical PFN 1 PFN 2 PFN 3 PFN n LFN 1 LFN 2 LFN 3 LFN n LFN 0 / Grid Unique ID RLSRepMeC A File

20 P.Kunszt C Replica location problem Replica Location Service: –maintains information on the physical location of files –maintains mapping between the logical identifier of data and all its physical instances –provides access to this information Given a unique logical identifier for some given data, determine the physical location of one or more physical instances of this data

21 P.Kunszt C RLS Requirements  Versioning & read only data –distinct versions of files can be uniquely identified –data published to the community are immutable  Size –scale to hundreds of replica sites, 50 x 10 8 LFNs, 500 x 10 8 PFNs  Performance –200 updates/second, average response time < 10ms  Security –knowledge and existence of private data must be protected –storage system protects integrity of data content  Consistency –view of all available PFNs not necessarily consistent  Reliability –no single point of failure, –local and global state decoupled, failure of remote component does not hinder access to local component

22 P.Kunszt C Giggle Framework Giggle: A Framework for Constructing Scalable Replica Location Services – Joint collaboration between WP2 and Globus – Paper submitted to SC2002 LRCs  Independent local state maintained in Local Replica Catalogues : LRCs RLIs  Unreliable collective state maintained in Replica Location Indices : RLIs  Soft state maintenance of RLI state –relaxed consistency in the RLI, full state information in LRC  Compression of soft states –compress LFN information based on knowledge of logical collections  Membership and partitioning information maintenance –RLS components change over time : failure, new components added –Service discovery and system policies

23 P.Kunszt C Local Replica Catalogue (LRC) Maintains replica information at a single site –Complete locally consistent record –Queries across multiple sites not supported Maintains mappings between LFNs and PFNs on associated storage systems –Coordinates its contents with those of the storage system Responds to the following queries : –Given an LFN, find the set of PFNS associated with that LFN –Given a PFN, find the set of LFNS associated with that PFN Supports authentication and authorisation when processing remote requests Periodically sends information about its state to the RLIs

24 P.Kunszt C Replica Location Index (RLI) Index structure needed to support queries across multiple sites One or more RLIs to map LFNs to LRCs –Structure w.r.t LRCs can be freely defined –redundancy, performance, scalability Geographical partitioning – all PFNs of a set of LRCs are indexed Namespace partitioning 1 – for load balancing purposes Namespace partitioning 2 – only LFNs adhering to a specified pattern are indexed for all LRCs –possibly not good for load balancing Many identical RLIs may be set up for load balancing

25 P.Kunszt C RLS Architecture (1) A 2 level RLS layout: The RLIs contain pointers to LRCs only. LRC RLI LRC Multiply indexed LRC for higher availability RLI indexing over the full namespace (all LRCs are indexed) RLI indexing over a subset of LRCs LRC indexed by only one RLI

26 P.Kunszt C RLS Architecture (2) A hierarchical RLS topology: The RLIs point to LRCs and RLIs RLI LRC RLI LRC Multiply indexed LRC for higher availability RLI indexing over the full namespace (all LRCs are indexed) RLI indexing over a subset of LRCs LRC indexed by only one RLI

27 P.Kunszt C Metadata Management and Security Project Spitfire 'Simple' Grid Persistency –Grid Metadata –Application Metadata –Unified Grid enabled front end to relational databases. Metadata Replication and Consistency Publish information on the metadata service Secure Grid Services Grid authentication, authorization and access control mechanisms enabled in Spitfire Modular design, reusable by other Grid Services

28 P.Kunszt C Spitfire Architecture OracleDB2PostGresMySQL Atomic RDBMS is always consistent No local replication of data Role-based authorization XSQL Servlet as one access mode for ‘simple’ web access Web/Grid Services Paradigm –SOAP interfaces –JDBC interface to RDBMS Pluggability and extensibility OracleLayerDB2LayerPGLayerMyLayer Local Spitfire Layer Connecting Layer Global Spitfire Layer SOAP

29 P.Kunszt C Security Mechanism Servlet Container SSLServletSocketFactory TrustManager Security Servlet Does user specify role? Map role to connection id Authorization Module HTTP + SSL Request + client certificate Yes Role Trusted CAs Is certificate signed by a trusted CA? No Has certificate been revoked? Revoked Certs repository Find default No Role repository Role ok? Connection mappings Translator Servlet RDBMS Request and connection ID Connection Pool

30 P.Kunszt C Status Integrated XSQL-Spitfire including security mechanisms XSQL version of a Replica MetaData Catalog –Schema is given –Roles are fixed –Queries are predefined –No SOAP, just XML over http Installation decoupled from Tomcat & MySQL Work on SOAP interface using axis Security mechanisms planned Spitfire & Security

31 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

32 P.Kunszt C WP2 Replication Services Replica Manager Replica Metadata Replica Location File Transfer Optimization Transaction Consistency Preprocessing Postprocessing Subscription Client Reptor Giggle RepMeC Optor GDMP

33 P.Kunszt C Plans Service-type Replica Manager integrated with Giggle and Replica Metadata Catalog Optimization module integrated Automated replication module designed and prototyped Replication & Optimization

34 P.Kunszt C Plans All functionality available Security fully integrated Installation works with more than just Tomcat and MySQL Test-suite completely implemented Working Replication Metadata Catalog using this Beta, integration with Replica Manager Spitfire & Security

35 P.Kunszt C Summary of Plans GDMP v3.0 will be the LAST release of GDMP as we know it. GDMP in the future will rely on the Replica Manager and provide the subscription based mirroring functionality. It will be implemented as a web service. The Replica Catalog will be replaced with the Giggle framework, jointly developed with Globus. The Replica Manager will take over most of GDMPs current functionality and will be a web service. We provide client APIs. All user interaction should go through the RM. Spitfire will have both an XML over http access method for static queries and a SOAP interface. Security infrastructure for all services in place as presented.

36 P.Kunszt C Outline WP2 Mandate, Tasks and Structure Achievements to date Task status –Replication –Metadata –Optimization –Security Plans Issues

37 P.Kunszt C Issues Getting everybody started, coordination with partners – one very late deliverable Technology challenges – evolving field, keeping on the top needs a lot of training effort of everybody Coordination within WP2 and US projects – not enough funds for CERN Very late arrival of and changes in manpower – additional training and supervision A lot of young members are working / have worked on their PhD Different agenda of user community delays development: continuous requests for support and enhancements of components that we want to phase out (GDMP) takes out development manpower, documentation suffers most.

38 P.Kunszt C Outlook Very ambitious programme, can we make it? Current workforce is very well motivated and has a very high level of interaction – see our mailinglist archives A lots of enhancements in inter-task coordination was done, further effort is necessary – workshops THANK YOU FOR YOUR ATTENTION