Data Management Expert Panel - WP2. WP2 Overview.

Slides:



Advertisements
Similar presentations
INFSO-RI Enabling Grids for E-sciencE EGEE and gLite Slides by: Erwin Laure EGEE Deputy Middleware Manager.
Advertisements

30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.
1 WP2: Data Management Paul Millar eScience All Hands Meeting September
WP2: Data Management Gavin McCance University of Glasgow November 5, 2001.
WP2: Data Management Gavin McCance University of Glasgow.
WP2 and GridPP UK Simulation W. H. Bell University of Glasgow EDG – WP2.
Data Management Expert Panel. RLS Globus-EDG Replica Location Service u Joint Design in the form of the Giggle architecture u Reference Implementation.
Andrew McNab - Manchester HEP - 2 May 2002 Testbed and Authorisation EU DataGrid Testbed 1 Job Lifecycle Software releases Authorisation at your site Grid/Web.
EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,
Andrew McNab - EDG Access Control - 14 Jan 2003 EU DataGrid security with GSI and Globus Andrew McNab University of Manchester
E-science grid facility for Europe and Latin America A Data Access Policy based on VOMS attributes in the Secure Storage Service Diego Scardaci.
Holding slide prior to starting show. Supporting Collaborative Working of Construction Industry Consortia via the Grid - P. Burnap, L. Joita, J.S. Pahwa,
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Services Abderrahman El Kharrim
N° 1 LCG EDG Data Management Catalogs in LCG James Casey LCG Fellow, IT-DB Group, CERN
Distributed Heterogeneous Data Warehouse For Grid Analysis
GGF Toronto Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.
DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.
UMIACS PAWN, LPE, and GRASP data grids Mike Smorul.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug
GRID job tracking and monitoring Dmitry Rogozin Laboratory of Particle Physics, JINR 07/08/ /09/2006.
GridPP9 – 5 February 2004 – Data Management DataGrid is a project funded by the European Union GridPP is funded by PPARC WP2+5: Data and Storage Management.
C Copyright © 2009, Oracle. All rights reserved. Appendix C: Service-Oriented Architectures.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
A Metadata Catalog Service for Data Intensive Applications Presented by Chin-Yi Tsai.
INFSO-RI Enabling Grids for E-sciencE gLite Data Management Services - Overview Mike Mineter National e-Science Centre, Edinburgh.
GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.
1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong
DataGrid is a project funded by the European Union CHEP 2003 – March 2003 – Next Generation Data Mgmt... – n° 1 James Casey CERN
ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.
QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.
PanDA Multi-User Pilot Jobs Maxim Potekhin Brookhaven National Laboratory Open Science Grid WLCG GDB Meeting CERN March 11, 2009.
1 WP2: Data Management Gavin McCance RAL Middleware Workshop 24 February 2003.
Author - Title- Date - n° 1 Partner Logo EU DataGrid, Work Package 5 The Storage Element.
Tools for collaboration How to share your duck tales…
Copyright © cs-tutorial.com. Overview Introduction Architecture Implementation Evaluation.
The Replica Location Service The Globus Project™ And The DataGrid Project Copyright (c) 2002 University of Chicago and The University of Southern California.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Replica Management Services in the European DataGrid Project Work Package 2 European DataGrid.
CLRC and the European DataGrid Middleware Information and Monitoring Services The current information service is built on the hierarchical database OpenLDAP.
CGW 04, Stripped replication for the grid environment as a web service1 Stripped replication for the Grid environment as a web service Marek Ciglan, Ondrej.
Overview of Privilege Project at Fermilab (compilation of multiple talks and documents written by various authors) Tanya Levshina.
EGEE User Forum Data Management session Development of gLite Web Service Based Security Components for the ATLAS Metadata Interface Thomas Doherty GridPP.
DGC Paris WP2 Summary of Discussions and Plans Peter Z. Kunszt And the WP2 team.
AliEn AliEn at OSC The ALICE distributed computing environment by Bjørn S. Nilsen The Ohio State University.
Glite. Architecture Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware Higher-Level Grid Services are supposed.
1 AHM, 2–4 Sept 2003 e-Science Centre GRID Authorization Framework for CCLRC Data Portal Ananta Manandhar.
JAliEn Java AliEn middleware A. Grigoras, C. Grigoras, M. Pedreira P Saiz, S. Schreiner ALICE Offline Week – June 2013.
Globus: A Report. Introduction What is Globus? Need for Globus. Goal of Globus Approach used by Globus: –Develop High level tools and basic technologies.
Ákos FROHNER – DataGrid Security n° 1 Security Group TODO
DGC Paris Spitfire A Relational DB Service for the Grid Leanne Guy Peter Z. Kunszt Gavin McCance William Bell European DataGrid Data Management.
MGRID Architecture Andy Adamson Center for Information Technology Integration University of Michigan, USA.
Rights Management for Shared Collections Storage Resource Broker Reagan W. Moore
1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.
VOX Project Status T. Levshina. 5/7/2003LCG SEC meetings2 Goals, team and collaborators Purpose: To facilitate the remote participation of US based physicists.
Site Authorization Service Local Resource Authorization Service (VOX Project) Vijay Sekhri Tanya Levshina Fermilab.
Building Preservation Environments with Data Grid Technology Reagan W. Moore Presenter: Praveen Namburi.
DataGrid Security Wrapup Linda Cornwall 4 th March 2004.
ACGT Architecture and Grid Infrastructure Juliusz Pukacki ‏ EGEE Conference Budapest, 4 October 2007.
Preservation Data Services Persistent Archive Research Group Reagan W. Moore October 1, 2003.
Overview of the New Security Model Akos Frohner (CERN) WP8 Meeting VI DataGRID Conference Barcelone, May 2003.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
Grid Services for Digital Archive Tao-Sheng Chen Academia Sinica Computing Centre
October 2014 HYBRIS ARCHITECTURE & TECHNOLOGY 01 OVERVIEW.
Jean-Philippe Baud, IT-GD, CERN November 2007
Manuel Brugnoli, Elisa Heymann UAB
Gavin McCance University of Glasgow GridPP2 Workshop, UCL
GGF OGSA-WG, Data Use Cases Peter Kunszt Middleware Activity, Data Management Cluster EGEE is a project funded by the European.
Introduction to Data Management in EGI
Presentation transcript:

Data Management Expert Panel - WP2

WP2 Overview

Talk Outline u Introduction to EDG Work Package 2 u WP2 Services: Design and Interactions n Spitfire n Replication Services n Grid Simulation n Security u Status

Grid middleware architecture hourglass Current Grid architectural functional blocks: OS, Storage & Network services Basic Grid Services High Level Grid Services Grid Application Services (LCG) Common application layer CMSATLASCMSLHCb Specific application layer GLOBUS 2.2 EU DataGrid WP2

EU DataGrid WP2 Data Management Work Package Responsible for u Transparent data location and secure access u Wide-area replication u Data access optimization u Metadata access NOT responsible for (but partially relying on other WPs for) u Data storage u Proper Relational Database bindings u Remote I/O u Security infrastructure

WP2 Service Paradigms u Choice of technology: n Web Services (servers implemented in Java) s Tomcat, Oracle 9iAS n Interface definitions are exposed in WSDL n Client stubs for many languages (Java, C, C++) s Axis, gSOAP (auto-generated) n Persistent service data in Relational Databases s MySQL, Oracle u Modularity n Modular service design for pluggability and extensibility n No vendor specific lock-ins u Evolvable n Easy adaptation to OGSA foreseen, based on the same technology n Largely independent of underlying OS, RDBMS

Spitfire: Grid-enabling RDBMS u Capabilities: n Simple Grid enabled front-end to any type of local or remote RDBMS through secure web services n Sample generic RDBMS methods may easily be customized with little additional development n Web browser integration n GSI authentication n Hooks in place for local authorization u Status: current release version 2.1 n Used by EU DataGrid Earth Observation and Biomedical applications. n Not currently suitable for the retrieval of LARGE result sets

Storage Element Replication Services: Basic Functionality Replica Manager Replica Location Service Replica Metadata Catalog Storage Element Files have replicas stored at many Grid sites on Storage Elements. Each file has a unique Grid ID (GUID). Replica Location Service maps the GUID to the multiple physical locations of that file. Users may assign aliases to the GUIDs. These are kept in the Replica Metadata Catalog. The Replica Manager provides atomicity for file operations, assuring consistency of SE and catalog contents.

Storage Element Higher Level Replication Services Replica Manager Replica Location Service Replica Optimization Service Replica Metadata Catalog SE Monitor Network Monitor Storage Element The Replica Manager calls the Replica Optimization service to find the best replica based upon network and SE monitoring information.

Storage Element Interactions with other Grid components Replica Manager Replica Location Service Replica Optimization Service Replica Metadata Catalog SE Monitor Network Monitor Information Service Resource Broker User Interface or Worker Node Storage Element Virtual Organization Membership Service Applications and users will manage data only through the Replica Manager - either directly or via the Resource Broker. Management calls should never go directly to the SE.

Grid Simulation (OptorSim) u Standalone data-centric Grid simulation used to develop and evaluate replication strategies - Grid2003: Simulation e.g. of CMS spring 2002 testbed s of jobs, ~100 GB files (50 GB capacity SEs). - Access patterns based on measured CDF analysis jobs. - To add in measured background traffic on network links

Security: Infrastructure for Java- based Web Services u Trust Manager n Mutual client-server authentication using GSI (ie PKI X509 certificates) for all WP2 services n Supports everything transported over SSL u Authorization Manager n Supports coarse grained authorization: Mapping user DN -> role -> attribute n Fine grained authorization through policies, role and attribute maps n Web-based Admin interface for managing the authorization policies and tables u Status: n Fully implemented, authentication is enabled on the service level n Delegation implementation currently being developed n Authorization (using VOMS) currently being integrated with WP2 services.

WP2 Status u Current Status n All components are available now n Initial tests show that expected performance can be met n Need proper testing in a real user environment – EDG2; LCG1 n Good results from OptorSim. Work continuing. u Work-plan for next release n Full integration of the authorization module. n Replica Location Index. n See James talk.