CGW 04, Stripped replication for the grid environment as a web service1 Stripped replication for the Grid environment as a web service Marek Ciglan, Ondrej.

Slides:

Advertisements

Similar presentations

GridSAM Overview Grid Job S ubmission A nd M onitoring Service What is GridSAM? Funded by the OMII Managed Programme (Started in Sept, 04) Client Perspective.

Advertisements

21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson.

1 WP2: Data Management Paul Millar eScience All Hands Meeting September

HEPiX Edinburgh 28 May 2004 LCG les robertson - cern-it-1 Data Management Service Challenge Scope Networking, file transfer, data management Storage management.

HEPiX GFAL and LCG data management Jean-Philippe Baud CERN/IT/GD.

Data Management Expert Panel - WP2. WP2 Overview.

Data Management Expert Panel. RLS Globus-EDG Replica Location Service u Joint Design in the form of the Giggle architecture u Reference Implementation.

ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.

Plateforme de Calcul pour les Sciences du Vivant SRB & gLite V. Breton.

Lightweight Preservation Environment Gary Jackson.

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Services Abderrahman El Kharrim

Application of GRID technologies for satellite data analysis Stepan G. Antushev, Andrey V. Golik and Vitaly K. Fischenko 2007.

Grid Programming Environment (GPE) Grid Summer School, July 28, 2004 Ralf Ratering Intel - Parallel and Distributed Solutions Division (PDSD)

Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.

Towards an agent integrated speculative scheduling service L á szl ó Csaba L ő rincz, Attila Ulbert, Tam á s Kozsik, Zolt á n Horv á th ELTE, Department.

UMIACS PAWN, LPE, and GRASP data grids Mike Smorul.

Northwestern University 2007 Winter – EECS 443 Advanced Operating Systems The Google File System S. Ghemawat, H. Gobioff and S-T. Leung, The Google File.

EU 2nd Year Review – Jan – WP9 WP9 Earth Observation Applications Demonstration Pedro Goncalves :

The Data Replication Service Ann Chervenak Robert Schuler USC Information Sciences Institute.

ALICE data access WLCG data WG revival 4 October 2013.

NAREGI WP4 (Data Grid Environment) Hideo Matsuda Osaka University.

Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.

Polish Infrastructure for Supporting Computational Science in the European Research Space QoS provisioning for data-oriented applications in PL-Grid D.

Grid Data Management A network of computers forming prototype grids currently operate across Britain and the rest of the world, working on the data challenges.

GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.

INFSO-RI Enabling Grids for E-sciencE The US Federation Miron Livny Computer Sciences Department University of Wisconsin – Madison.

The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Dataset Caitlin Minteer & Kelly Clynes.

Miguel Branco CERN/University of Southampton Enabling provenance on large-scale e-Science applications.

File and Object Replication in Data Grids Chin-Yi Tsai.

Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

MAGDA Roger Jones UCL 16 th December RWL Jones, Lancaster University MAGDA  Main authors: Wensheng Deng, Torre Wenaus Wensheng DengTorre WenausWensheng.

Part Four: The LSC DataGrid Part Four: LSC DataGrid A: Data Replication B: What is the LSC DataGrid? C: The LSCDataFind tool.

Enabling Grids for E-sciencE Introduction Data Management Jan Just Keijser Nikhef Grid Tutorial, November 2008.

Optimizing Live Migration of Virtual Machines across Wide Area Networks using Integrated Replication and Scheduling Sumit Kumar Bose, Unisys Scott Brock,

INFSO-RI Enabling Grids for E-sciencE OGSA DAI Data Access and Integration Marek Ciglan Institute of Informatics, Slovac Academy.

Replica Management Services in the European DataGrid Project Work Package 2 European DataGrid.

LEGS: A WSRF Service to Estimate Latency between Arbitrary Hosts on the Internet R.Vijayprasanth 1, R. Kavithaa 2,3 and Raj Kettimuthu 2,3 1 Coimbatore.

11/5/2001WP5 UKHEPGRID1 WP5 Mass Storage UK HEPGrid UCL 11th May Tim Folkes, RAL

T3 analysis Facility V. Bucard, F.Furano, A.Maier, R.Santana, R. Santinelli T3 Analysis Facility The LHCb Computing Model divides collaboration affiliated.

Interactive Workflows Branislav Šimo, Ondrej Habala, Ladislav Hluchý Institute of Informatics, Slovak Academy of Sciences.

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

DGC Paris WP2 Summary of Discussions and Plans Peter Z. Kunszt And the WP2 team.

Replica Management Kelly Clynes. Agenda Grid Computing Globus Toolkit What is Replica Management Replica Management in Globus Replica Management Catalog.

Introduction to The Storage Resource.

Managing Learning Objects in Large Scale Courseware Authoring Studio Ivo Marinchev, Ivo Hristov Institute of Information Technologies Bulgarian Academy.

Managing Data DIRAC Project. Outline  Data management components  Storage Elements  File Catalogs  DIRAC conventions for user data  Data operation.

Optimizing Live Migration of Virtual Machines across Wide Area Networks using Integrated Replication and Scheduling Sumit Kumar Bose, Unisys Scott Brock,

Bookkeeping Tutorial. 2 Bookkeeping content  Contains records of all “jobs” and all “files” that are produced by production jobs  Job:  In fact technically.

NorduGrid plans and questions for gLite Marko Niinimaki, NorduGrid 3 rd EGEE meeting Athens, April 2005.

Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation.

Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.

10 May 2001WP6 Testbed Meeting1 WP5 - Mass Storage Management Jean-Philippe Baud PDP/IT/CERN.

EGI-Engage Data Services and Solutions Part 1: Data in the Grid Vincenzo Spinoso EGI.eu/INFN Data Services.

Data Management The European DataGrid Project Team

An approach to Web services Management in OGSA environment By Shobhana Kirtane.

EGEE is a project funded by the European Union under contract IST Information and Monitoring Services within a Grid R-GMA (Relational Grid.

Rights Management for Shared Collections Storage Resource Broker Reagan W. Moore

1 Data Management for Internet Backplane Protocol by Tang Ming Assoc/Prof. Francis Lee School of Computer Engineering, Nanyang Technological University,

1 DIRAC Data Management Components A.Tsaregorodtsev, CPPM, Marseille DIRAC review panel meeting, 15 November 2005, CERN.

Grid Services for Digital Archive Tao-Sheng Chen Academia Sinica Computing Centre

System Software Laboratory Databases and the Grid by Paul Watson University of Newcastle Grid Computing: Making the Global Infrastructure a Reality June.

Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations.

GFAL Grid File Access Library

The Data Grid: Towards an architecture for Distributed Management

Joseph JaJa, Mike Smorul, and Sangchul Song

Data Management in Release 2

Grid Data Replication Kurt Stockinger Scientific Data Management Group Lawrence Berkeley National Laboratory.

gLite The EGEE Middleware Distribution

Presentation transcript:

CGW 04, Stripped replication for the grid environment as a web service1 Stripped replication for the Grid environment as a web service Marek Ciglan, Ondrej Habala, Ladislav Hluchý Institute of informatics Slovak Academy of Sciences

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service2 Overview Replication in Grid environment Principles of stripped replication (SR) method Optimization of stripped replication Prototype Implementation as a Web Service Experimental Results Future Work

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service3 Replication in Grid environment Creation of multiple copies of single data source across Grid infrastructure Replication increases data availability RLS - Replica Location Service Grid monitoring services – network monitoring

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service4 Replication in Grid environment Storage Element 1 File 1 Storage Element 2 Storage Element 3

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service5 Replication in Grid environment Storage Element 1 File 1 Storage Element 2 Storage Element 3 File 1

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service6 Replication in Grid environment Storage Element 1 File 1 Storage Element 2 Storage Element 3 File 1

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service7 Replication in Grid environment Storage Element 1 File 1 Storage Element 2 Storage Element 3 File 1

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service8 Stripped Replication - Principles Transfer from multiple Grid sites, in parallel Transfer only a portion of file from each Storage Element (SE) Different file portions (stripes) are obtained from different SEs Parallel transfer increases replication speed If SR is not managed properly, process could be time consuming Optimization of SR management is required

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service9 Stripped Replication - Optimization Replicated data source Replica 1Replica 2Replica 3

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service10 Stripped Replication - Optimization Replicated data source Replica 1Replica 2Replica 3 Replica 1Replica 2Replica 3

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service11 Stripped Replication - Optimization Replica 1Replica 2Replica 3

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service12 Stripped Replication - Optimization Replica 1Replica 2Replica 3 Replica 1Replica 2Replica 3

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service13 Stripped Replication - Optimization Replica 1Replica 2Replica 3 Replica 1Replica 2Replica 3 Replica 1Replica 2Replica 3

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service14 SR Prototype Implementation Java programming language CoG 1.2 API (GridFTP interface) Integrated with EDG Replica Location Service EDG RLS API (RLS interface) File Chunks – basic data units for transfer Implemented as a Web Service ( motivation :OGSA, WSRF)

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service15 Service Workflow Stripped Replication Service LFN Get GUID

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service16 Service Workflow LFN Get GUID Replica Metadata Catalog Stripped Replication Service

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service17 Service Workflow LFN Get GUID Replica Metadata Catalog Stripped Replication Service Get PFNs

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service18 Service Workflow LFN Get GUID Replica Metadata Catalog Get PFNs Local Replica Catalog Stripped Replication Service

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service19 Service Workflow LFN Get GUID Replica Metadata Catalog Get PFNs Local Replica Catalog Stripped Replication Algorithm Stripped Replication Service

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service20 Service Workflow LFN Get GUID Replica Metadata Catalog Get PFNs Local Replica Catalog Stripped Replication Algorithm GridFTP Site 1 GridFTP Site N... Stripped Replication Service

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service21 Service Workflow LFN Get GUID Replica Metadata Catalog Get PFNs Local Replica Catalog Stripped Replication Algorithm GridFTP Site 1 GridFTP Site N... Register Replica Stripped Replication Service

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service22 Properties of Stripped Replication Parallel transfer from multiple sites increases replication process speed Proposed optimization does not use network monitoring services SR adapts to varying nature of network load SR optimally distributes network load

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service23 Experimental Results Motivation test case –File size 223.9Mb –Best replica transfer with standard replication tool (EDG rm) sec –Stripped replication (2 replicas) – 405 sec (43 %) –Stripped replication (3 replicas) – 209 sec (71 %) Average time saving –2 replicas – 37% time saving –3 replica – 55% time saving

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service24 Future Work Implementation refinement –Add logging functionality –Refine error states handling Evaluation of SR integration in Grid projects

Stripped Replication for Grids CGW 04, Stripped replication for the grid environment as a web service25 Thank you for your attention !