The Storage Resource Broker and the NGS Slides from Wayne Schroeder, SDSC and Peter Berrisford, RAL

Induction to Grid Computing and the NGS 2 Policy for re-use This presentation can be re-used for academic purposes. However if you do so then please let training- know. We need to gather statistics of re-use: no. of events, number of people trained. Thank you!!training-

Induction to Grid Computing and the NGS 3 Acknowledgements This tutorial selects slides from several sources, specifically from talks given by Wayne Schroeder (SDSC) and Peter Berrisford (RAL)

Induction to Grid Computing and the NGS 4 Goal Introduce use of the SRB for distributed file management on the NGS –This is the focus of the practical that follows NOTE: –Wayne Schroeder: SRB does so much, people tend to learn subsets and are often unaware of useful features –So explore further! For a full SRB tutorial, see:

Induction to Grid Computing and the NGS 5 What is SRB? (1 of 3) The SDSC Storage Resource Broker (SRB) is client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and accessing unique or replicated data objects. SRB, in conjunction with the Metadata Catalog (MCAT), provides a way to access data sets and resources based on their logical names or attributes rather than their names and physical locations.

Induction to Grid Computing and the NGS 6 What is SRB? (2 of 3) The SDSC SRB system is a comprehensive distributed data management solution, with features to support the management, collaborative (and controlled) sharing, publication, and preservation of distributed data collections. The SRB also serves as middleware via a rich set of APIs available to higher-level applications and by providing a management layer on top of a wide variety of storage systems.

Induction to Grid Computing and the NGS 7 What is SRB? (3 of 3) The SRB is an integrated solution which includes: –a logical namespace, –interfaces to a wide variety of storage systems, –high performance data movement (including parallel I/O), –fault-tolerance and fail-over, –WAN-aware performance enhancements (bulk operations), –storage-system-aware performance enhancements ('containers' to aggregate files), –metadata ingestion and queries (a MetaData Catalog (MCAT)), –user accounts, groups, access control, audit trails, GUI administration tool –data management features, replication –user tools (including a Windows GUI tool (inQ), a set of SRB Unix commands, and Web (mySRB)), and APIs (including C, C++, Java, and Python). SRB Scales Well (many millions of files, terabytes) Supports Multiple Administrative Domains / MCATs (srbZones) And includes SDSC Matrix: SRB-based data grid workflow management system to create, access and manage workflow process pipelines.

Induction to Grid Computing and the NGS 8 SRB Projects Digital Libraries –UCB, Umich, UCSB, Stanford,CDL –NSF NSDL - UCAR / DLESE NASA Information Power Grid Astronomy –National Virtual Observatory –2MASS Project (2 Micron All Sky Survey) Particle Physics –Particle Physics Data Grid (DOE) –GriPhyN –SLAC Synchrotron Data Repository Medicine –Digital Embryo (NLM) Earth Systems Sciences –ESIPS –LTER Persistent Archives –NARA –LOC Neuro Science & Molecular Science –TeleScience/NCMIR, BIRN –SLAC, AfCS, … Over 90 Tera Bytes in 16 million files

Induction to Grid Computing and the NGS 9 SRB Scalability

Induction to Grid Computing and the NGS 10 What is SRB? Storage Resource Broker (SRB) is a software product developed by the San Diego Supercomputing Centre (SDSC). Allows users to access files and database objects across a distributed environment. Actual physical location and way the data is stored is abstracted from the user Allows the user to add user defined metadata describing the scientific content of the information

Induction to Grid Computing and the NGS 11 How SRB Works MCAT Database MCAT Server SRB A Server SRB B Server SRB Client a b cd e f g 4 major components: –The Metadata Catalogue (MCAT) –The MCAT-Enabled SRB Server –The SRB Storage Server –The SRB Client

Induction to Grid Computing and the NGS 12 SRB Client Tools Provide a user interface to send requests to the SRB server. 4 main interfaces: –Command line (S-Commands) –MS Windows (InQ) –Web based (MySRB). –Java (JARGON) Web Services (MATRIX)

Induction to Grid Computing and the NGS 13 Planned Deployment on NGS Online Replication SRB Server SRB RAL Resource Driver MCAT SRB Oxford Resource Driver Database RAL User Disk Farm Disk Farm MCAT RAL DB1 DB n Database Manchester DB1 DB n SRB Server MCAT MCAT Manchester Failover link Disk Farm SRB HPCX Resource Driver Disk Farm SRB Leeds Resource Driver Disk Farm

Induction to Grid Computing and the NGS 14 Summary SRB provides NGS users with – a virtual filesystem –Accessible from all core nodes and from the UI / desktop –(will provide) redundancy – mirrored catalogue server –Replica files –Support for application metadata associated with files

Induction to Grid Computing and the NGS 15 Tutorial Overview Use of the Scommands –Commands for unix based access to srb –Strong analogy to unix file commands Accessing files from multiple (two) sites

Induction to Grid Computing and the NGS 16 Getting Started Launch two putty connections to pub-234 –One for commands run on pub-234 –One for connecting to grid-data.rl.ac.uk and running commands from there Open browser and follow link from Follow the instructions from there. Your SRB name is the same as your account on pub-234