The Storage Resource Broker and the NGS Slides from Wayne Schroeder, SDSC and Peter Berrisford, RAL

2 Acknowledgements This tutorial selects slides from several sources, specifically from talks given by Wayne Schroeder (SDSC) and Peter Berrisford (RAL)

3 Goal Introduces a subset of the SRB functionality –Distributed file management for NGS users –This is the focus of the practical that follows NOTE: SRB has many facets! –Wayne Schroeder: SRB does so much, people tend to learn subsets and are often unaware of useful features –So explore further! For a full SRB tutorial, see:

4 What is SRB? (1 of 3) The SDSC Storage Resource Broker (SRB) is client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and accessing unique or replicated data objects. SRB, in conjunction with the Metadata Catalog (MCAT), provides a way to access data sets and resources based on their logical names or attributes rather than their names and physical locations.

5 What is SRB? (2 of 3) The SDSC SRB system is a comprehensive distributed data management solution, with features to support the management, collaborative (and controlled) sharing, publication, and preservation of distributed data collections. The SRB also serves as middleware via a rich set of APIs available to higher-level applications and by providing a management layer on top of a wide variety of storage systems.

6 What is SRB? (3 of 3) The SRB is an integrated solution which includes: –a logical namespace, –interfaces to a wide variety of storage systems, –high performance data movement (including parallel I/O), –fault-tolerance and fail-over, –WAN-aware performance enhancements (bulk operations), –storage-system-aware performance enhancements ('containers' to aggregate files), –metadata ingestion and queries (a MetaData Catalog (MCAT)), –user accounts, groups, access control, audit trails, GUI administration tool –data management features, replication –user tools (including a Windows GUI tool (inQ), a set of SRB Unix commands, and Web (mySRB)), and APIs (including C, C++, Java, and Python). SRB Scales Well (many millions of files, terabytes) Supports Multiple Administrative Domains / MCATs (srbZones) And includes SDSC Matrix: SRB-based data grid workflow management system to create, access and manage workflow process pipelines.

7 SRB Projects Digital Libraries –UCB, Umich, UCSB, Stanford,CDL –NSF NSDL - UCAR / DLESE NASA Information Power Grid Astronomy –National Virtual Observatory –2MASS Project (2 Micron All Sky Survey) Particle Physics –Particle Physics Data Grid (DOE) –GriPhyN –SLAC Synchrotron Data Repository Medicine –Digital Embryo (NLM) Earth Systems Sciences –ESIPS –LTER Persistent Archives –NARA –LOC Neuro Science & Molecular Science –TeleScience/NCMIR, BIRN –SLAC, AfCS, … Over 90 Tera Bytes in 16 million files

8 SRB Scalability

9 What is SRB? Storage Resource Broker (SRB) is a software product developed by the San Diego Supercomputing Centre (SDSC). Allows users to access files and database objects across a distributed environment. Actual physical location and way the data is stored is abstracted from the user Allows the user to add user defined metadata describing the scientific content of the information

10 How SRB Works MCAT Database MCAT Server SRB A Server SRB B Server SRB Client a b cd e f g 4 major components: –The Metadata Catalogue (MCAT) –The MCAT-Enabled SRB Server –The SRB Storage Server –The SRB Client

11 SRB Client Tools Provide a user interface to send requests to the SRB server. 4 main interfaces: –Command line (S-Commands) –MS Windows (InQ) –Web based (MySRB). –Java (JARGON) Web Services (MATRIX)

12 Planned Deployment on NGS Online Replication SRB Server SRB RAL Resource Driver MCAT SRB Oxford Resource Driver Database RAL User Disk Farm Disk Farm MCAT RAL DB1 DB n Database Manchester DB1 DB n SRB Server MCAT MCAT Manchester Failover link Disk Farm SRB HPCX Resource Driver Disk Farm SRB Leeds Resource Driver Disk Farm

13 Summary SRB provides NGS users with – a virtual filesystem –Accessible from all core nodes and from the UI / desktop –(will provide) redundancy – mirrored catalogue server –Replica files –Support for application metadata associated with files

SRB Tutorial Guy Warner NeSC Training Team

15 Overview Use of the Scommands –Commands for unix based access to srb –Strong analogy to unix file commands Accessing files from multiple (two) sites

16 Getting Started Launch two putty connections to pub-234 –One for commands run on pub-234 –One for connecting to grid-data.rl.ac.uk and running commands from there Open browser at Follow the instructions from there. Your srb name is the same as your account on pub- 234

17 SRB practical