Presentation is loading. Please wait.

Presentation is loading. Please wait.

Enabling Grids for E-sciencE EGEE-II INFSO-RI The Development of SRM interface for SRB Fu-Ming Tsai Academia Sinica Grid Computing.

Published byWalter Woods Modified over 7 years ago

Similar presentations

Presentation on theme: "Enabling Grids for E-sciencE EGEE-II INFSO-RI The Development of SRM interface for SRB Fu-Ming Tsai Academia Sinica Grid Computing."— Presentation transcript:

1 Enabling Grids for E-sciencE www.eu-egee.org EGEE-II INFSO-RI-031688 The Development of SRM interface for SRB Fu-Ming Tsai Academia Sinica Grid Computing sary357@twgrid.org

2 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 2 What is SRM? SRM is an unique interface for accessing diffident backend storages for diffident middleware. Easy to develop applications to adapt different backend storages. Provide space and file management on the storage system. SRM is the web service interface and the implementation usually depends on the backend storage technology.

3 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 3 Why SRM(Storage resource management)? Storage Elements (SE) can use different type of technologies –CASTOR, dCache, DPM, BeStMan,...,etc. –DRM (Disk Resource Manager)/TRM (Tape Resource Manager) /HRM (Hierarchical Resource Manager) Grid middleware needs to access files with an uniform interface –Manage storage resources –Not a file transfer protocol

4 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 4 SRM features Provides space management Provides an uniform access interface Manages DRM/Tape/HRM Does not transfer files itself. Manage the life time of file

5 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 5 What is SRB (Storage Resource Broker) ? Developed by San Diego Supercomputer Center A distributed file management system (Data Grid), based on a client-server architecture. Allows users to access files seamlessly across a distributed environment, based upon their attributes rather than just their names or physical locations. It replicates, syncs, archives, and connects heterogeneous resources in a logical and abstracted manner.

6 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 6 SRB Physical Structure Oracle ClientSRB Server Oracle RDBMS SRB Vault @ location B SRB Vault @ location B SRB Vault @ location D SRB Server Storage Space Storage Driver SRB Server Storage Space Storage Driver SRB Server Storage Space Storage Driver User @ location X

7 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 7 SRM for SRB – Our Implementation

8 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 8 Cache server (+gridftp server) Architecture Overview Core Cache repository SRB+DSI File catalog SRB/gridftp Gridftp/management API SRM API File transfer (gridftp) Web Service Data server management Users/applications

9 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 9 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Non MES+DSI Cache repository MES Server (GSI enabled) SRB storage space Non MES File catalog (AMGA) Put a file (SURL) SURL->Path Where to put the file? Non MES (+DSI) Non MES (+DSI)

10 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 10 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Non MES+DSI Cache repository MES Server (GSI enabled) SRB storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) Return TURL Transfer to TURL Server name and path name

11 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 11 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Storage space Cache repository MES Server (+GSI enabled) Storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) Upload a file(gridftp)

12 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 12 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Storage space Cache repository MES Server (GSI enabled) Storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) put file done SRBInfoServer InfoServer SURL -> Path Update status of this file Update all resource information Query InfoServe r Query SRBInfoServer Query InfoServer

13 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 13 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Storage space Cache repository MES Server (GSI enabled) Storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) The Status of PutDone SRBInfoServer InfoServer Return the Status of PutDone Update status of each resource Resource info Logical Resource info Resource info

14 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 14 Support Flexible File/Space Types SRM system has a caching mechanism and has to take care of SRM issues like file lifetime, space management, …,etc. –Volatile space –Durable space –Permanent space In our implementation –Volatile space & Durable space  In the temporary storage space. –Permanent space  In the SRB env.

15 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 15 Checking Disk Status How to get the disk usage of the space? –Need to know the free and used space on cache server and SRB server –SRB does not provide the mechanism to monitor resource usage –We need to know the usage  Space management In our implementation –InfoServer:  Deployed on cache server and non-Mcat enabled SRB server –SRBInfoServer:  Deployed on Mcat-enabled SRB server

16 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 16 Naming Mapping between SURL, TURL, and file name –Different naming space on cache server and SRB server  /XXX/YYY/ZZZ: cache server  /Zone/home/UserAccount.UserDomain/: SRB server  How to map them?? In our implementation –Use AMGA to provide uniform naming space  /XXX/YYY/ZZZ –How to map to TURL  On cache server Use the mounted point gotten from InfoServer and use this mounted point to store user’s data, then reply TURL  On SRB server Try to login and get user’s home collection. Use user’s home collection to store user’s data, then reply TURL.

17 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 17 Typical Use Case

18 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 18 Testbed deployment Hostname: fct01.grid.sinica.edu.tw The end point: httpg://fct01.grid.sinica.edu.tw:8443/axis/services/srm Info: Cache server (gridftp server) and SRM interface Hostname: t-ap20.grid.sinica.edu.tw Info: SRB server (SRB-DSI installed) User InterfaceUser Interface SURL Gridftp/manage ment commands Return some information TURL Host information File information Hostname:t-ap51.grid.sinca.edu.tw Info: AMGA server

19 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 19 Demo Test the functions that we have implemented. Access data in –Cache server –SRB server Use LFC (just a simulated LFC host) to get SURL and use this SURL to connect to SRM server, then get TURL back.

20 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 20 Demo(1) Test the following functions –SrmPing –SrmGetTransferProtocols –SrmCheckPermission –SrmGetPermission –SrmMkdir –SrmRmdir –SrmPrepareToPut –SrmPutDone –SrmPrepareToGet –SrmReleaseFiles –SrmRm

21 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 21 Demo(2) Put/get data in/from cache env through SRM Put/get data in/from cache SRB env through SRM

22 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 22 User needs to store data in SE (from a UI) 1.Create a new LFN entry in LFC, return a SURL. 2.srmPrepateToPut (SURL) 3.Transfer the file 4.srmPutDone (SURL) Demo(3)

23 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 23 User needs to retrieve (onto the UI) data stored into SE 1.Query the file catalog to retrieve the SURL from the LFN. 2.srmPrepateToGet (SURL) 3.Transfer the file (read) 4.srmReleaseFile (SURL) Demo(3)

24 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 24 Progress What we have done: –We have finished the parts about authorizing users, web service interface (just interface) and gridftp deployment, and SRB-DSI, and some functions like directory functions, permission functions,..,etc. What we are doing: –Now, we are focusing on the implementation of our core (data transfer functions and space management).

25 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 25 Future Plan Short term –Implement interface compatible SRM version 2.2. Mid-term –Interact and test with other data management systems such as Castor, dCache, …,etc. Long-term –Interoperate with gLite.  Hope glite users who join VO can access files and resources in SRB and vice versa.

26 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 26 Future Plan

27 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 27 Future Plan SRB SRM

28 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 28 References SRM-SRB interface project wiki page –http://lists.grid.sinica.edu.tw/apwiki/SRM-SRBhttp://lists.grid.sinica.edu.tw/apwiki/SRM-SRB SRM working group: –http://sdm.lbl.gov/srm-wg/ SRB: –http://www.sdsc.edu/srb/index.php/Main_Page Globus: –http://www.globus.org CoG: –http://wiki.cogkit.org/index.php/Main_Page Axis: –http://ws.apache.org/axis/

29 EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE 29 Thanks for your attention

Download ppt "Enabling Grids for E-sciencE EGEE-II INFSO-RI The Development of SRM interface for SRB Fu-Ming Tsai Academia Sinica Grid Computing."

Similar presentations

EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org www.glite.org The gLite middleware distribution OSG Consortium Meeting Seattle, 21-23.

EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,
Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.

Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Services Abderrahman El Kharrim

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Services Abderrahman El Kharrim
Applying Data Grids to Support Distributed Data Management Storage Resource Broker Reagan W. Moore Ian Fisk Bing Zhu University of California, San Diego.

Applying Data Grids to Support Distributed Data Management Storage Resource Broker Reagan W. Moore Ian Fisk Bing Zhu University of California, San Diego.
Magda – Manager for grid-based data Wensheng Deng Physics Applications Software group Brookhaven National Laboratory.

Magda – Manager for grid-based data Wensheng Deng Physics Applications Software group Brookhaven National Laboratory.
Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug 2009 1.

Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug
Jan. 2006 Storage Resource Broker Managing Distributed Data in a Grid A discussion of a paper published by a group of researchers at the San Diego Supercomputer.

Jan Storage Resource Broker Managing Distributed Data in a Grid A discussion of a paper published by a group of researchers at the San Diego Supercomputer.
INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org gLite Data Management Services - Overview Mike Mineter National e-Science Centre, Edinburgh.

INFSO-RI Enabling Grids for E-sciencE gLite Data Management Services - Overview Mike Mineter National e-Science Centre, Edinburgh.
A. Sim, CRD, L B N L 1 Oct. 23, 2008 BeStMan Extra Slides.

A. Sim, CRD, L B N L 1 Oct. 23, 2008 BeStMan Extra Slides.
INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.

INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org Data Grid Services/SRB/SRM & Practical Hai-Ning Wu Academia Sinica Grid Computing.

EGEE-II INFSO-RI Enabling Grids for E-sciencE Data Grid Services/SRB/SRM & Practical Hai-Ning Wu Academia Sinica Grid Computing.
EGEE-III INFSO-RI- 222667 Enabling Grids for E-sciencE www.eu-egee.org The Medical Data Manager : the components Johan Montagnat, Romain Texier, Tristan.

EGEE-III INFSO-RI Enabling Grids for E-sciencE The Medical Data Manager : the components Johan Montagnat, Romain Texier, Tristan.
Enabling Grids for E-sciencE www.eu-egee.org ENEA and the EGEE project gLite and interoperability Andrea Santoro, Carlo Sciò Enea Frascati, 22 November.

Enabling Grids for E-sciencE ENEA and the EGEE project gLite and interoperability Andrea Santoro, Carlo Sciò Enea Frascati, 22 November.
Data Management The GSM-WG Perspective. Background SRM is the Storage Resource Manager A Control protocol for Mass Storage Systems Standard protocol:

Data Management The GSM-WG Perspective. Background SRM is the Storage Resource Manager A Control protocol for Mass Storage Systems Standard protocol:
The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.

The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.
EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE middleware Data Management in gLite.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE middleware Data Management in gLite.
D C a c h e Michael Ernst Patrick Fuhrmann Tigran Mkrtchyan d C a c h e M. Ernst, P. Fuhrmann, T. Mkrtchyan Chep 2003 Chep2003 UCSD, California.

D C a c h e Michael Ernst Patrick Fuhrmann Tigran Mkrtchyan d C a c h e M. Ernst, P. Fuhrmann, T. Mkrtchyan Chep 2003 Chep2003 UCSD, California.
Author - Title- Date - n° 1 Partner Logo EU DataGrid, Work Package 5 The Storage Element.

Author - Title- Date - n° 1 Partner Logo EU DataGrid, Work Package 5 The Storage Element.
EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE middleware: gLite Data Management EGEE Tutorial 23rd APAN Meeting, Manila Jan.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE middleware: gLite Data Management EGEE Tutorial 23rd APAN Meeting, Manila Jan.
Andrew C. Smith – Storage Resource Managers – 10/05/05 Functionality and Integration Storage Resource Managers.

Andrew C. Smith – Storage Resource Managers – 10/05/05 Functionality and Integration Storage Resource Managers.

Similar presentations

Ads by Google