1. Grid Data Replication
Kurt Stockinger Scientific Data Management Group Lawrence Berkeley National Laboratory

2. File Management Motivation
Site A
Site B
Storage Element A
Storage Element B
File Transfer
File Management on the Grid
What file management functionalities are expected? The promise of the Grid is that the user files will be managed by the Grid services. The user’s files are identified by a name that is known to the user, the Logical File Name, as shown in this slide (File A,B,C,D,X,Y). A file may be replicated to many Grid sites, in the example shown on the slide, file A and B are available on both sites, C,D,X,Y are only available on one of the two sites.
In the following we will argue for the file management functionalities that are necessary for the DataGrid users (HEP, EO, BIO) in a Grid environment.
In order to make files available on more than one site, we need File Transfer services between sites. The Storage Element is defined (from the point of view of Data Management) as an abstract storage service that ‘speaks’ several file transfer protocols, for example FTP, HTTP, GridFTP, RFIO. The File Transfer service is a service offered by the Storage Element, the default being GridFTP for DataGrid.
File A
File X
File A
File C
File B
File Y
File B
File D

3. File Management Motivation
Replica Catalog: Map Logical to Site files
Site A
Site B
Storage Element A
Storage Element B
File Transfer
If files are available from more than one storage site (because they have been transferred using the File Transfer service), we need a service that keeps track of them. This service is the Replica Catalog, it contains the mappings of the Logical File Names to their Site File Names, i.e. the name by which the file can be accessed through the Storage Element in a given site.
File A
File X
File A
File C
File B
File Y
File B
File D

4. File Management Motivation
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Site A
Site B
Storage Element A
Storage Element B
File Transfer
Next, we need to provide a service that makes a choice between the available replicas based on a set of information available from the Grid: network bandwith, storage element access speed, etc.
File A
File X
File A
File C
File B
File Y
File B
File D

5. File Management Motivation
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Site A
Site B
Storage Element A
Storage Element B
File Transfer
Many file types need some kind of pre- and/or post-processing before and/or after transfer. These steps might be related to data extraction, conversion, encryption for pre-processing and importation, conversion, decryption, etc for post-prosessing. Also, as a Quality-of-Service step the file may be validated after transfer as a post-processing step (by checking its checksum, for example). This service should be customizable by the applications that use certain file types.
File A
File X
File A
File C
File B
File Y
File B
File D

6. File Management Motivation
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Storage Element A
Storage Element B
File Transfer
Automated replication based on a subscription model is also desirable especially for HEP, where the data coming from the LHC will have to be distributed automatically over the world. The subscription semantics may be defined through file name patterns, data locations, etc. This service should also be customizable by the Virtual Organizations setting it up.
File A
File X
File A
File C
File B
File Y
File B
File D

7. File Management Motivation
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Storage Element A
Storage Element B
File Transfer
There is another desirable automatic replication mechanism that should be available through a service, which is automated replication to balance the access load on ‘popular’ files. This service needs to store and analyze user access patterns on the files available through the Grid. Replication is triggered by rules that are applied to the usage patterns on the existing files.
File A
File X
File A
File C
File B
File Y
File B
File D

8. Replica Manager: ‘atomic’ replication operation single client interface orchestrator
File Management
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Storage Element A
Storage Element B
File Transfer
The user does not want to be burdened to contact each of these services directly, taking care of the correct order of operations and making sure that errors are caught and handled accordingly. This functionality is provided by the Replica Manager which acts as the single interface to all replication services. As an additional QoS functionality it should provide transactional integrity of the steps involved in the high-level usages of the underlying services. It should be able to gracefully recover interrupted transfers, catalog updates, pre- and post-processing steps, etc.
File A
File X
File A
File C
File B
File Y
File B
File D

9. Replica Manager: ‘atomic’ replication operation single client interface orchestrator
File Management
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Metadata: LFN metadata
Transaction information
Access patterns
Storage Element A
Storage Element B
File Transfer
A helper service to be used by the File Management services described until now is the metadata service. It should be able to store all metadata associated with Data Management, like attributes on LFNs, patterns for the load balancing service, transaction locks for the replica manager, etc.
File A
File X
File A
File C
File B
File Y
File B
File D

10. Security File Management
Replica Manager: ‘atomic’ replication operation single client interface orchestrator
File Management
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Security
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Metadata: LFN metadata
Transaction information
Access patterns
Storage Element A
Storage Element B
File Transfer
Across all of these services we need to impose the stamp of Security, which means that the services need to authenticate and authorize their clients. In order to provide a higher level of consistency between the replicas, we require the users to transfer all administrative rights on the files that they register through the Replica Manager. In order to administer their files they therefore will need to go through the Replica Manager again – administering the files directly through the Storage will be denied. This enables the Replica Manager to act for the user on the user’s behalf – automated replication would not be possible without this restriction.
File A
File X
File A
File C
File B
File Y
File B
File D

11. Interactions with other Grid Components
Virtual Organization
Membership Service
User Interface or
Worker Node
Resource Broker
Information Service
Replica Metadata
Catalog (RMC)
Replica Location
Service (RLS)
Replica Manager
Replica Optimization
Service (ROS)
Blue: WP2 services
Storage
Element
Storage
Element SE
Monitor
Network Monitor

12. Simplified Interaction Replica Manager - SRM
Catalog 1 2
Replica Manager
client 6
SRM 3 4 5 6
Storage
The Client asks a catalog to provide the location of a file
The catalog responds with the name of an SRM
The client asks the SRM for the file
The SRM asks the storage system to provide the file
The storage system sends the file to the client through the SRM or
directly

13. User Interfaces for Data Management
Users are mainly referred to use the interface of the Replica Manager client:
Management commands
Catalog commands
Optimization commands
File Transfer commands
The services RLS, RMC and ROS provide additional user interfaces
Mainly for additional catalog operations (in case of RLS, RMC)
Additional server administration commands
Should mainly be used by administrators
Can also be used the check the availability of a service RM
RLS
RMC
ROS

14. The Replica Manager Interface – Management Commands
copyAndRegisterFile args: source, dest, lfn,  protocol, streams
Copy a file into grid-aware storage and register the copy in the Replica Catalog as an atomic operation.
replicateFile args: source/lfn, dest,  protocol, streams
Replicate a file between grid-aware stores and register the replica in the Replica Catalog as an atomic operation.
deleteFile args: source/seHost, all  
Delete a file from storage and unregister it.
Example
edg-rm --vo=cms copyAndRegisterFile
file:/home/bob/analysis/data5.dat
-d lxshare0384.cern.ch
This and the next slide show the API functions of the Replica Manager(RM), to be used by clients and applications.
Main command line options:
-l: logical file name
-s: source file name
-d: destination file name
-c: config file for the replica catalog
-e: verbose error output (important for testing!)

15. The Replica Manager Interface – Catalog Commands
registerFile args: source, lfn
Register a file in the Replica Catalog that is already stored on a Storage Element.
unregisterFile args: source, guid
Unregister a file from the Replica Catalog.
listReplicas args: lfn/surl/guid
List all replicas of a file.
registerGUID args: surl, guid
Register an SURL with a known GUID in the Replica Catalog.
listGUID args: lfn/surl
Print the GUID associated with an LFN or SURL.
This and the next slide show the API functions of the Replica Manager(RM), to be used by clients and applications.
Main command line options:
-l: logical file name
-s: source file name
-d: destination file name
-c: config file for the replica catalog
-e: verbose error output (important for testing!)

16. The Replica Manager Interface – Optimization Commands
listBestFile args: lfn/guid,  seHost
Return the 'best' replica for a given logical file identifier.
getBestFile args: lfn/guid,  seHost,  protocol, streams
Return the storage file name (SFN) of the best file in terms of network latencies.
getAccessCost args: lfn/guid[], ce[], protocol[]
Calculates the expected cost of accessing all the files specified by logicalName from each Computing Element host specified by ceHosts.
This and the next slide show the API functions of the Replica Manager(RM), to be used by clients and applications.
Main command line options:
-l: logical file name
-s: source file name
-d: destination file name
-c: config file for the replica catalog
-e: verbose error output (important for testing!)

17. The Replica Manager Interface – File Transfer Commands
copyFile args: soure, dest
Copy a file to a non-grid destination.
listDirectory args: dir
List the directory contents on an SRM or a GridFTP server.
This and the next slide show the API functions of the Replica Manager(RM), to be used by clients and applications.
Main command line options:
-l: logical file name
-s: source file name
-d: destination file name
-c: config file for the replica catalog
-e: verbose error output (important for testing!)

18. Conclusions and Further Information
The second generation Data Management services have been designed and implemented based on the Web Service paradigm
Flexible, extensible service framework
Deployment choices : robust, highly available commercial products supported (eg. Oracle) as well as open-source (MySQL, Tomcat)
First experiences with these services show that their performance meets the expectations
Further information / documentation: