Enabling Grids for E-sciencE EGEE-II INFSO-RI The Development of SRM interface for SRB Fu-Ming Tsai Academia Sinica Grid Computing

EGEE-II INFSO-RI Enabling Grids for E-sciencE 2 What is SRM? SRM is an unique interface for accessing diffident backend storages for diffident middleware. Easy to develop applications to adapt different backend storages. Provide space and file management on the storage system. SRM is the web service interface and the implementation usually depends on the backend storage technology.

EGEE-II INFSO-RI Enabling Grids for E-sciencE 3 Why SRM(Storage resource management)? Storage Elements (SE) can use different type of technologies –CASTOR, dCache, DPM, BeStMan,...,etc. –DRM (Disk Resource Manager)/TRM (Tape Resource Manager) /HRM (Hierarchical Resource Manager) Grid middleware needs to access files with an uniform interface –Manage storage resources –Not a file transfer protocol

EGEE-II INFSO-RI Enabling Grids for E-sciencE 4 SRM features Provides space management Provides an uniform access interface Manages DRM/Tape/HRM Does not transfer files itself. Manage the life time of file

EGEE-II INFSO-RI Enabling Grids for E-sciencE 5 What is SRB (Storage Resource Broker) ? Developed by San Diego Supercomputer Center A distributed file management system (Data Grid), based on a client-server architecture. Allows users to access files seamlessly across a distributed environment, based upon their attributes rather than just their names or physical locations. It replicates, syncs, archives, and connects heterogeneous resources in a logical and abstracted manner.

EGEE-II INFSO-RI Enabling Grids for E-sciencE 6 SRB Physical Structure Oracle ClientSRB Server Oracle RDBMS SRB location B SRB location B SRB location D SRB Server Storage Space Storage Driver SRB Server Storage Space Storage Driver SRB Server Storage Space Storage Driver location X

EGEE-II INFSO-RI Enabling Grids for E-sciencE 7 SRM for SRB – Our Implementation

EGEE-II INFSO-RI Enabling Grids for E-sciencE 8 Cache server (+gridftp server) Architecture Overview Core Cache repository SRB+DSI File catalog SRB/gridftp Gridftp/management API SRM API File transfer (gridftp) Web Service Data server management Users/applications

EGEE-II INFSO-RI Enabling Grids for E-sciencE 9 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Non MES+DSI Cache repository MES Server (GSI enabled) SRB storage space Non MES File catalog (AMGA) Put a file (SURL) SURL->Path Where to put the file? Non MES (+DSI) Non MES (+DSI)

EGEE-II INFSO-RI Enabling Grids for E-sciencE 10 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Non MES+DSI Cache repository MES Server (GSI enabled) SRB storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) Return TURL Transfer to TURL Server name and path name

EGEE-II INFSO-RI Enabling Grids for E-sciencE 11 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Storage space Cache repository MES Server (+GSI enabled) Storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) Upload a file(gridftp)

EGEE-II INFSO-RI Enabling Grids for E-sciencE 12 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Storage space Cache repository MES Server (GSI enabled) Storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) put file done SRBInfoServer InfoServer SURL -> Path Update status of this file Update all resource information Query InfoServe r Query SRBInfoServer Query InfoServer

EGEE-II INFSO-RI Enabling Grids for E-sciencE 13 Architecture Overview (cont.) Web Service Core Data server management Cache server (+gridftp server) Storage space Cache repository MES Server (GSI enabled) Storage space Non MES (+DSI) Non MES (+DSI) File catalog (AMGA) The Status of PutDone SRBInfoServer InfoServer Return the Status of PutDone Update status of each resource Resource info Logical Resource info Resource info

EGEE-II INFSO-RI Enabling Grids for E-sciencE 14 Support Flexible File/Space Types SRM system has a caching mechanism and has to take care of SRM issues like file lifetime, space management, …,etc. –Volatile space –Durable space –Permanent space In our implementation –Volatile space & Durable space  In the temporary storage space. –Permanent space  In the SRB env.

EGEE-II INFSO-RI Enabling Grids for E-sciencE 15 Checking Disk Status How to get the disk usage of the space? –Need to know the free and used space on cache server and SRB server –SRB does not provide the mechanism to monitor resource usage –We need to know the usage  Space management In our implementation –InfoServer:  Deployed on cache server and non-Mcat enabled SRB server –SRBInfoServer:  Deployed on Mcat-enabled SRB server

EGEE-II INFSO-RI Enabling Grids for E-sciencE 16 Naming Mapping between SURL, TURL, and file name –Different naming space on cache server and SRB server  /XXX/YYY/ZZZ: cache server  /Zone/home/UserAccount.UserDomain/: SRB server  How to map them?? In our implementation –Use AMGA to provide uniform naming space  /XXX/YYY/ZZZ –How to map to TURL  On cache server Use the mounted point gotten from InfoServer and use this mounted point to store user’s data, then reply TURL  On SRB server Try to login and get user’s home collection. Use user’s home collection to store user’s data, then reply TURL.

EGEE-II INFSO-RI Enabling Grids for E-sciencE 17 Typical Use Case

EGEE-II INFSO-RI Enabling Grids for E-sciencE 18 Testbed deployment Hostname: fct01.grid.sinica.edu.tw The end point: httpg://fct01.grid.sinica.edu.tw:8443/axis/services/srm Info: Cache server (gridftp server) and SRM interface Hostname: t-ap20.grid.sinica.edu.tw Info: SRB server (SRB-DSI installed) User InterfaceUser Interface SURL Gridftp/manage ment commands Return some information TURL Host information File information Hostname:t-ap51.grid.sinca.edu.tw Info: AMGA server

EGEE-II INFSO-RI Enabling Grids for E-sciencE 19 Demo Test the functions that we have implemented. Access data in –Cache server –SRB server Use LFC (just a simulated LFC host) to get SURL and use this SURL to connect to SRM server, then get TURL back.

EGEE-II INFSO-RI Enabling Grids for E-sciencE 20 Demo(1) Test the following functions –SrmPing –SrmGetTransferProtocols –SrmCheckPermission –SrmGetPermission –SrmMkdir –SrmRmdir –SrmPrepareToPut –SrmPutDone –SrmPrepareToGet –SrmReleaseFiles –SrmRm

EGEE-II INFSO-RI Enabling Grids for E-sciencE 21 Demo(2) Put/get data in/from cache env through SRM Put/get data in/from cache SRB env through SRM

EGEE-II INFSO-RI Enabling Grids for E-sciencE 22 User needs to store data in SE (from a UI) 1.Create a new LFN entry in LFC, return a SURL. 2.srmPrepateToPut (SURL) 3.Transfer the file 4.srmPutDone (SURL) Demo(3)

EGEE-II INFSO-RI Enabling Grids for E-sciencE 23 User needs to retrieve (onto the UI) data stored into SE 1.Query the file catalog to retrieve the SURL from the LFN. 2.srmPrepateToGet (SURL) 3.Transfer the file (read) 4.srmReleaseFile (SURL) Demo(3)

EGEE-II INFSO-RI Enabling Grids for E-sciencE 24 Progress What we have done: –We have finished the parts about authorizing users, web service interface (just interface) and gridftp deployment, and SRB-DSI, and some functions like directory functions, permission functions,..,etc. What we are doing: –Now, we are focusing on the implementation of our core (data transfer functions and space management).

EGEE-II INFSO-RI Enabling Grids for E-sciencE 25 Future Plan Short term –Implement interface compatible SRM version 2.2. Mid-term –Interact and test with other data management systems such as Castor, dCache, …,etc. Long-term –Interoperate with gLite.  Hope glite users who join VO can access files and resources in SRB and vice versa.

EGEE-II INFSO-RI Enabling Grids for E-sciencE 26 Future Plan

EGEE-II INFSO-RI Enabling Grids for E-sciencE 27 Future Plan SRB SRM

EGEE-II INFSO-RI Enabling Grids for E-sciencE 28 References SRM-SRB interface project wiki page – SRM working group: – SRB: – Globus: – CoG: – Axis: –

EGEE-II INFSO-RI Enabling Grids for E-sciencE 29 Thanks for your attention