EGEE-II INFSO-RI Enabling Grids for E-sciencE Data Grid Services/SRB/SRM & Practical Hai-Ning Wu Academia Sinica Grid Computing

Enabling Grids for E-sciencE EGEE-II INFSO-RI Outlines Introduction Characteristics of data grid Storage Resource Management (SRM) Storage Resource Broker (SRB) –SRB Practical Summary

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Storage in Large Scales Historically data has been STORED rather than MANAGED The amount of data grows so rapidly that traditional storage architectures are no longer suitable Data are distributed in multiple types of source – hard to integrate data and increase the barriers between users and storage systems

Enabling Grids for E-sciencE EGEE-II INFSO-RI Challenges of Data Storage Large scales of data Distributed storage via network Heterogeneous data resources Management data with efficiency and safety Long-term preservation

Enabling Grids for E-sciencE EGEE-II INFSO-RI The Solution: Data Grid Data virtualization –Manipulates data in high level –Hides details in low level Provides a uniform interface to access the distributed data storage systems

Enabling Grids for E-sciencE EGEE-II INFSO-RI Virtualization Data virtualization Trust virtualization Data grids are used to manage shared collections that are distributed across multiple sites and multiple storage systems

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Grid - The Idea Data Grid Data found Request for Data Client Users

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Grid - The Idea Data found Request for Data Client Users Details are hidden. The data grid system finds out where the data are located. Data Grid System

Enabling Grids for E-sciencE EGEE-II INFSO-RI The Goals of Data Grid Automating preservation process Mitigating risk of data loss Supporting retrieval and access A uniform interface to manage digital entities stored in any type of storage system Providing access through wide varieties of access mechanisms Support of technology evolution

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Grid Transparencies Find data without knowing the identifier –Descriptive attributes Access data without knowing the location –Logical name space Access data without knowing the type of storage –Storage repository abstraction Retrieve data using your preferred API –Access abstraction Provide transformations for any data collection –Data behavior abstraction

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Grid Components Federated client-server architecture –Servers can talk to each other independently of the client Infrastructure independent naming –Logical names for users, resources, files, applications Collective ownership of data –Collection-owned data, with infrastructure independent access control lists Context management –Record state information in a metadata catalog from data grid services such as replication Abstractions for dealing with heterogeneity

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Grid Architecture Unix Shell Java, NT Browsers OAI, WSDL OGSA HTTP Archives - Tape, HPSS, ADSM, UniTree, DMF, CASTOR,ADS Databases DB2, Oracle, Sybase, SQLserver,Postgres, mySQL, Informix File Systems Unix, NT, Mac OSX Application ORB Standard Storage System Operations Interface Standard Database Interface Databases DB2, Oracle, Sybase, Postgres, mySQL, Informix C, C++, Java Libraries Logical Name Space Management Latency Management Digital Component Transport Metadata Transport Consistency & Metadata Management / Authorization-Authentication Audit Linux I/O DLL / Python, Perl Federation Management

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRM The Data Grid Interface for EGEE Grid Middleware

Enabling Grids for E-sciencE EGEE-II INFSO-RI gLite Services

Enabling Grids for E-sciencE EGEE-II INFSO-RI SE Storage Element –The Storage Element is the service which allows a user or an application to store data –Data Channel Protocols  File Transfer and File I/O

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRM (Storage Resource Management) What is SRM? –SRM is a protocol to manage storage resources (It is NOT a file access protocol!) –Provides an uniform interface for computing applications and client users to heterogeneous storage elements –Does not transfer files itself –Provides space management –Manage the life time of file

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRM & Grid

Enabling Grids for E-sciencE EGEE-II INFSO-RI Grid files Grid Files –Files in the Grid can be referred by different names:  Logical File Name (LFN) : An alias created by a user to refer to some item of data. For example, /grid/gilda/gridcamp/testFile.txt  Grid Unique IDentifier (GUID) : A non-human-readable unique identifier for an item of data. For example, 37afd0cc-c53b-4795-a873-6a9dde35a9cc  Site URL (SURL) : The location of an actual piece of data on a storage system. For example, srm://dpm01.grid.sinica.edu.tw/dpm/grid.sinica.edu.tw/home/twgrid/generated/ /file4c4a5a6f- 878d-4ef3-a73d-941ae  Transport URL (TURL) : Temporary locator of a replica + access protocol: understood by a SE. For example, gsiftp://dpm01.grid.sinica.edu.tw/dpm01.grid.sinica.edu.tw:/path1/twgrid/ /file4c4a5a6f-878d-4ef3-a73d-941ae –While the GUIDs and LFNs identify a file irrespective of its location, the SURLs and TURLs contain information about where a physical replica is located, and how it can be accessed.

Enabling Grids for E-sciencE EGEE-II INFSO-RI LFC File Catalogue (LFC) –The mappings between LFNs, GUIDs and SURLs are kept in a File Catalogue service, while the files themselves are stored in Storage Elements. –The only file catalogue officially supported in WLCG/EGEE is the LCG File Catalogue (LFC). Mapping by the “LFC” catalogue server

Enabling Grids for E-sciencE EGEE-II INFSO-RI Upload a file to a SE CASE 1 User needs to store data in SE (from a UI) 1.Create a new LFN entry in LFC, return a SURL. 2.srmPrepateToPut (SURL) 3.Transfer the file 4.srmPutDone (SURL)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Upload a file to a SE CASE 2 Application needs to store data in SE (from a WN) 1.Create a new LFN entry in LFC, return a SURL. 2.srmPrepateToPut (SURL) 3.Transfer the file 4.srmPutDone (SURL)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Download files from a SE CASE 3 User needs to retrieve (onto the UI) data stored into SE 1.Query the file catalog to retrieve the SURL from the LFN. 2.srmPrepateToGet (SURL) 3.Transfer the file (read) 4.srmReleaseFile (SURL)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Download files from a SE CASE 4 Application needs to copy data locally (into the WN) and use them. 1.Query the file catalog to retrieve the SURL from the LFN. 2.srmPrepateToGet (SURL) 3.Transfer the file (read) 4.srmReleaseFile (SURL)

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Storage Resource Broker

Enabling Grids for E-sciencE EGEE-II INFSO-RI Storage Resource Broker Developed at San Diego Supercomputer Center A distributed file management system (Data Grid), based on a client-server architecture A uniform interface to heterogeneous data storage resources, Based upon their attributes rather than just their names or physical locations Support many data storage systems Provide various types of client interfaces on different platforms

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Physical Structure Oracle ClientSRB Server Oracle RDBMS SRB location A SRB location B SRB location D SRB Server Storage Space Storage Driver SRB Server Storage Space Storage Driver SRB Server Storage Space Storage Driver location X MCAT-Enabled Server

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Pratical - inQ Download inQ from Unzip inQ350.zip Execute inQ.exe

Enabling Grids for E-sciencE EGEE-II INFSO-RI inQ – Login Name: srbusr+your number Host: tap07.grid.sinica.edu.tw Domain: ASGC Port: 6833 Authorization: ENCRYPT1 Password: The same as your user name

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Client Tool - inQ

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Demonstrations Use InQ to upload, download, remove files. Use Scommands to upload, download, remove files. –Sinit: log in SRB system  Syntax: Sinit –Sls: list directory content  Syntax: Sls –Sput: upload a file to the SRB server  Syntax: Sput filename –Sget: download a file from the SRB server  Syntax: Sget filename –Srm: remove a file stored in SRB server  Syntax: Srm filename –Sreplicate: to replicate data to another resource  Syntax: Sreplicate filename –Sexit: log out SRB system  Syntax: Sexit

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Grid Applications Digital Archiving –Long-term preservation –Heterogeneous backup Digital Library –Data sharing Scientific Computing

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Use Case Build Data Grid Management System –Data Grid services in Academia Sinica –NDAP cross-organization data backup project

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Data Grid Services in Academia Sinica (1) Objective –To provide Grid services for long-term preservation and unified data access Data Collection Status –File size: ~ 60 TB –File count: ~ 3.5 Million

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRB Data Grid Service in Academia Sinica(2)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Academia Sinica (AS) National Palace Museum (NPM) National Taiwan University (NTU) National Museum of History (NMH) Academia Historica (DRNH) National Central Library (NCL) National Museum of Natural Science (NMNS) Taiwan Historica (TH) NDAP Partners For Long-term Data Preservation

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data grid for NDAP LTP service

Enabling Grids for E-sciencE EGEE-II INFSO-RI Summary Data grids provides a new solution for large-scale storage with the following features: –Distributed data storage –Efficient and safe management of data –A uniform interface to heterogeneous systems –Flexibility to new storage technology

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRM & SRB SRM –Used in gLite middleware –A uniform interface between different SEs and grid middleware SRB –Developed by SDSC –Support many backend storage systems –Widely used data grid software

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRM & SRB SRM and SRB cannot interoperate unless they have a standard to communicate Constructing a bridge between SRM and SRB so that –Integrate SRB into the gLite environment –Bind resources from the two important data grid systems –This project is currently developed by ASGC

Enabling Grids for E-sciencE EGEE-II INFSO-RI SRM & SRB

Enabling Grids for E-sciencE EGEE-II INFSO-RI iRODS A next generation data grid system after SRB developed by SDSC A rule-oriented data grid system More flexibility for data management Current version: iRODS 1.0

Enabling Grids for E-sciencE EGEE-II INFSO-RI iRODS Workshop Time – Tue 8 April 2008 Location – 2nd Conference Room, 3F For more information, please check on ISGC 2008 Website

Enabling Grids for E-sciencE EGEE-II INFSO-RI References [1] Use Cases on Data Services, Fu-Ming Tsai [2] Building Preservation Environments with Data Grid Technology, R. Moore [3] EGEE Middleware Architecture and Planning (Release 1)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Thanks for your attentions!