1. Production Data Grids SRB - iRODS Storage Resource Broker Reagan W. Moore moore@sdsc.edu http://www.sdsc.edu/srb

2. Topics Production data grids Architecture and installation challenges Production challenges Interoperability challenges (federation) Applications Data grids - sharing Digital libraries - publication Persistent archive - preservation Real-time sensor data - collection Cyberinfrastructure - analysis

3. BaBar High-Energy Physics Stanford Linear Accelerator Palo Alto, CA IN2P3 Lyon, France A functioning international Data Grid for high-energy physics Manchester-SDSC mirror Moved over 300 TBs of data Increasing to 5 TBs per day

4. Architecture Challenges Infrastructure heterogeneity Storage in file systems, archives, ORBs Choice of database for metadata catalog Network devices Management of firewalls, private virtual networks, load levelers Network latency Geographic distance between storage locations

5. Installation Choices Infrastructure heterogeneity Provision of drivers for each type of storage system or database Porting of APIs for each preferred access mechanism Network devices Establishment of range of ports for access through firewall Server-initiated parallel I/O and bulk operations Network latency Master-slave metadata catalogs Federation across multiple independent data grids

6. Data Grid Using a Data Grid – in Abstract Ask for data User asks for data from the data grid Data delivered The data is found and returned Where & how details are hidden

7. Data Grid Management Data grids integrate multiple system components Application level client software Federation software Data grid servers Data grid metadata catalog Security infrastructure Storage systems Database catalog Network A failure in any of the systems is viewed as a failure of the data grid

8. Operation Challenges Data grids provide mechanisms to analyze all types of infrastructure failure Integrity checks Authenticity checks System logs Data grids provide mechanisms to manage all types of infrastructure failure Replication of data and metadata Synchronization of replicas Federation of data grids Server rebooting and server maintenance mode

9. Operation Procedures Periodic system administration Manage integrity checks on data Manage audit trails Manage consistency checks on collections Manage synchronization of replicas Manage deletion of files (empty trash can) Track all errors and reported data losses Manage upgrades to new versions of the data grid servers Operational tasks for each data grid Add servers for new storage systems Add new users Respond to user questions Modify access controls on collections and storage Restart data grid servers as needed Identify problems with storage systems Respond to installation questions Integrate user interfaces with data grid

10. Automation of Management Tasks integrated Rule-Oriented Data System - iRODS Express management policies as rules that control the execution of micro-services Micro-service is a standard operation performed on a remote storage system Manage persistent state information that describes outcome of the micro-service Persistent Metadata catalog stores state information Virtualize the management policies Logical name space for rules Logical name space for micro-services Logical name space for state information First release in December 2006

11. iRODS - integrated Rule-Oriented Data System Resources Client InterfaceAdmin Interface Metadata Modifier Module Config Modifier Module Rule Modifier Module Consistency Check Module Confs Rule Base Metadata Persistent Repository Engine Rule Curren t State Rule Invoker Micro Service Modules Resource-based Micro-services Micro Service Modules Metadata-based Micro-services Service Manager Consistency Check Module Consistency Check Module

12. Interoperation Virtualization Management of federation with other data grid technologies Define micro-service that executes the protocols required by the alternate data grid Define rule for when this micro-service is executed (link to explicit storage location) Separately manage state information from application of this micro-service iRODS enables encapsulation of the rules, access mechanisms, and state information needed for interoperation with other data grids

13. Federation Between Data Grids Data Collection B Data Access Methods (Web Browser, DSpace, OAI-PMH) Data Grid Logical resource name space Logical user name space Logical file name space Logical persistent state name space Logical rule name space Logical micro-service name space Data Collection A Access controls and consistency constraints on cross registration of logical name spaces Data Grid Logical resource name space Logical user name space Logical file name space Logical persistent state name space Logical rule name space Logical micro-service name space

14. OGF Data Grid Federation

15. For More Information Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.sdsc.edu/srb/