1. The EU DataGrid Data Management
The European DataGrid Project Team

2. EDG Tutorial Overview Information Service Workload Management Services
Data Management Services
Networking
Information Service
Fabric Management

3. Overview Data Management Issues Main Components EDG Replica Catalog
EDG Replica Manager
GDMP
In this lecture you will learn about data management issues within a Data Grid and which are the main components to cope with these issues.
Subsequently, four data management tools which are currently deployed on the EDG testbed are presented.

4. Data Management Issues
On the grid the user user has access to different kinds of computing and data services which are connected via the Internet.
If you submit a job you typically don’t know in advance where it might end. The input data needed by a job might be on any storage element.

5. Data Management Issues
Of course, you might steer a job to a location where the data is available, but what if you need several data that is on different places?
Similarly, when your job is finished – where should you put the data such that it is easy accessibly by yourself and others in subsequent applications?

6. Data Management Tools Tools for On EDG Testbed you have Locating data
Copying data
Managing and replicating data
Meta Data management
On EDG Testbed you have
EDG Replica catalog
globus-url-copy (GridFTP)
EDG Replica Manager
Grid Data Mirroring Package (GDMP)
Spitfire
To cope with these issues you need several tools that allow to find data on the grid, to copy data, to replicate data (I.e. to produce exact copies to get better locality), and, ideally, high level data management tools that hide most of the complexity. In addition, applications may store additional meta data with their files which of course also needs to be managed.
On the EDG Testbed we currently provide for this:
A replica catalog that stores physical locations of files, I.e. Physical File Names together with a logical identifier, a Logical File Name;
Globus tools (globus-url-copy which uses the GridFTP protocol) for secure file transfer;
GDMP a data mirroring package that allows to automatically provide data on places that are interested in this data;
the EDG Replica Manager that allows to replicate files while keeping the replica catalog consistent;
and Spitfire, a frontend to grid enabled data bases for managing meta-data (this tool is not covered in this tutorial).

7. EDG Replica Catalog Based upon the Globus LDAP Replica Catalog
Stores LFN/PFN mappings and additional information (e.g. filesize):
Physical File Name (PFN): host + full path & and file name
Logical File Name (LFN): logical name that may be resolved to PFNs
LFN : PFN = 1 : n
Only files on storage elements may be registered
Each VO has a specific storage dir on an SE
Example PFN: lxshare0222.cern.ch/flatfiles/SE1/iteam/file1.dat host storage dir
LFN must be full path of file starting from storage dir LFN of above PFN: file1.dat
The EDG Replica Catalog is based upon the RC provided by Globus. It is essentially an LDAP server that stores LFN/PFN mappings together with additional information, such as filesize.
Note that only files stored on SEs may be stored in the catalog. Moreover, there is a restriction in filenames: every replica must have the same filename starting from a VO specific storage directory on the SE. This storage directory may be retrieved from the information services:
The LFN for a file must be the full path and filename starting from the storage directory.

8. EDG Replica Catalog API and command line tools
addLogicalFileName
getLogicalFileName
deleteLogicalFileName
getPhysicalFileName
addPhysicalFileName
deletePhysicalFileName
addLogicalFileAttribute
getLogicalFileAttribute
deleteLogicalFileAttribute
0/node85.html
Main command line options:
-l: logical file name
-p: physical file name
-n: attribute name
-v: attribute value
-c: config file name
-d: don’t show all output
-C: clear text password authorization (used currently in most catalog. –C is required!)

9. globus-url-copy Low level tool for secure copying
globus-url-copy <protocol>://<source file> \ <protocol>://<destination file>
Main Protocols:
gsiftp – for secure transfer, only available on SE and CE
file – for accessing files stored on the local file system on e.g. UI, WN
globus-url-copy file://`pwd`/file1.dat \ gsiftp://lxshare0222.cern.ch/ \ flatfiles/SE1/EDGTutorial/file1.dat
globus-url-copy is a low level tool for secure copying. It should only be used if just a copy without registering the file in the RC is desired. Otherwise, higher level tools as we discuss now should be used.

10. The EDG Replica Manager
Extends the Globus replica manager
Only client side tool
Allows replication (copy) and registering of files in RC
works with LDAP based RC and RLS (see future directions)
Keeps RC consistent with stored data.
Uses GDMP’s staging interface to stage to MSS
The edg replica manager may be used to replicate data and keeping the replica catalog in sync. It is a prototype developed based upon the globus replica manager in order to get experience for the ongoing development of an intelligent replica manager, called Reptor (cf. Lecture on Future Directions).

11. The Replica Manager APIs
(un)registerEntry(LogicalFileName lfn,
FileName source)
Replica Catalogue operations only - no file transfer
copyFile(FileName source,
FileName destination,
String protocol)
allows for third-party transfer
transfer between:
two StorageElements or
ComputingElement and Storage Element
Space management policies under development
all tools support parallel streams for file transfers
This and the next slide show the API functions of the Replica Manager(RM), to be used by clients and applications.
Main command line options:
-l: logical file name
-s: source file name
-d: destination file name
-c: config file for the replica catalog
-e: verbose error output (important for testing!)

12. The Replica Manager APIs
copyAndRegisterFile(LogicalFileName lfn,
FileName source,
FileName destination,
String protocol)
third-party transfer but :
files can only be registered in Replica Catalogue if destination PFN contains a valid SE (i.e. needs to be registered in the RC)!
replicateFile(LogicalFileName lfn,
deleteFile(LogicalFileName lfn,
FileName source)

13. Genius file replication

14. Genius file replication

15. Genius file replication

16. Genius file replication

17. Genius file delete

18. Genius file delete

19. Genius file delete

20. experience will directly be used in DataGrid
originally based on CMS requirements for replicating Objectivity files for High Level Trigger studies
production prototype project for evaluating Grid technologies (especially Globus)
experience will directly be used in DataGrid
input also for PPDG and GriPhyN
GDMP is a data mirroring package that was originally developed as a production prototype for evaluating grid technologies, in particular Globus, in context of CMS trigger studies.
GDMP has been integrated into EDG software and the experiences gained with it are exploited in ongoing EDG replication projects.

21. Overview of Components
Globus Replica Catalogue
GDMP client
GDMP is a client/server architecture where an GDMP server is running on every SE. GDMP servers may also communicate with each other and have access to a common RC.
A GDMP server is able to server multiple VOs with different replica catalogs.
Site1
Site3
Site2

22. Subscription Model
All the sites that subscribe to a particular site get notified whenever there is an update in its catalog.
Site 1
Site 2
Subscriber
list
Subscriber
list
subscribe
subscribe
GDMP works according to a subscription/notification scheme. A site that is interested in some particular data subscribes to the site where the data is produced and will subsequently be notified whenever an update is published on the provider site.
Site 3

23. Export / Import Catalogue
Export Catalog
information about the new files produced .
is published
Import Catalog
information about the files which have been published by other sites but not yet transferred locally
As soon as the file is transferred locally, it is removed from the import catalogue.
Possible to pull the information about new files into your import catalogue.
Site 1
Site 2
export
catalog
export
catalog
1)register, publish
new files
1) get info about
new files
import
catalog
The subscription/notification model works with help of 2 kinds of catalogs: an export and an import catalog.
The export catalog is published on the provider site and contains information on new files the provider wants to publish. The import catalog is automatically updated with new information available in any export catalog a site is subscribed to. Once a file referenced in the import catalog has been actually transferred, the corresponding entry is removed.
If the import catalog is unreachable, for instance due to network problems, it might miss updates from sites it is subscribed to. In such a case, an active pull may be used to update the import catalog.
3) delete files
Site 3
2) transfer files
2) transfer files

24. Usage gdmp_ping Ping a GDMP server and get its status
gdmp_host_subscribe
first thing to be done by a site
gdmp_register_local_file
Registers a file in local file catalogue but NOT in Replica Catalogue (RC)
gdmp_publish_catalogue
send information of newly created files to subscribed hosts (no real data transfer) – update RC
gdmp_replicate_get - gdmp_replicate_put
get/put all the files from the import catalogue – update RC
gdmp_remove_local_file
Delete a local file and update RC
gdmp_get_catalogue
Get remote catalogue contents – for error recovery
For detailed information on these commands as well as the command line options refer to the GDMP users guide at:

25. Site 2 Site 5 Site 1 Site 3 Site 4 Using GDMP
Register all files in a directory at site 1
gdmp_register_local_file –d /data/files
Site 2
Site 5
Site 1
/data/files/file1
/data/files/file2 …
Site 3
Site 4
Data produced at site 1 to be replicated to other sites

26. Site 5 Site 2 Site 1 Site 3 Site 4 Using GDMP 2
Start with subscription
gdmp_host_subscribe –r <HOST> -p <PORT>
Site 5
Site 2
gdmp_host_subscribe
gdmp_host_subscribe
Site 1
Subscriber
list
gdmp_host_subscribe
Site 3
Site 4

27. Site 5 Site 2 Site 1 Site 3 Site 4 Using GDMP 3
Publish new files…can combine with filtering
gdmp_publish_catalogue (might use filter option)
Import
catalog
Import
catalog
Site 5
Site 2
Export
catalog
Site 1
Subscriber
list
gdmp_publish_catalogue
Site 3
Import
catalog
Site 4

28. Site 5 Site 2 Site 1 Site 3 Site 4 Using GDMP 4
Poll for change in catalog (pull model)…can combine with filtering…also used for error recovery.
gdmp_get_catalogue –host <HOST>
Import
catalog
Import
catalog
Site 2
Site 5
Export
catalog
Site 1
Subscriber
list
gdmp_get_catalogue
Site 3
Site 4
Import
catalog
Import
catalog

29. Site 5 Site 2 Site 1 Site 3 Site 4 Using GDMP 5
Transfer files…can use the progress meter
gdmp_replicate_get
get_progress_meter…produces a progress.log.
replica.log has all files already transferred.
Import
catalog
Import
catalog
Site 2
Site 5
gdmp_replicate_get
gdmp_replicate_get
Export
catalog
Site 1
Subscriber
list
gdmp_replicate_get
Site 3
Site 4
Import
catalog
Import
catalog

30. GDMP vs. EDG Replica Manager
Replicates sets of files
Replication between SEs only
Mass storage interface
File size as logical attribute
Subscription model
Event notification
CRC file size check
Support for Objectivity
Replica Manager
Replicates single files
Replication between SEs, UI or CE to SE.
Uses GDMP’s Mass Storage interface at the SE
In contrast to GDMP, the edg-replica manager only supports basic replication of single files, however, it is able to copy between all kinds of testbed machines.
For simple replication tasks, edg-replica-manager is the preferred tool; for complex replication tasks including replication of multiple files to multiple locations gdmp should be used.
client-server
client side only

31. File Management Summary
Site A
Site B
Storage Element A
Storage Element B
File Management on the Grid
What file management functionalities are expected? The promise of the Grid is that the user files will be managed by the Grid services. The user’s files are identified by a name that is known to the user, the Logical File Name, as shown in this slide (File A,B,C,D,X,Y). A file may be replicated to many Grid sites, in the example shown on the slide, file A and B are available on both sites, C,D,X,Y are only available on one of the two sites.
In the following we will argue for the file management functionalities that are necessary for the DataGrid users (HEP, EO, BIO) in a Grid environment.
In order to make files available on more than one site, we need File Transfer services between sites. The Storage Element is defined (from the point of view of Data Management) as an abstract storage service that ‘speaks’ several file transfer protocols, for example FTP, HTTP, GridFTP, RFIO. The File Transfer service is a service offered by the Storage Element, the default being GridFTP for DataGrid.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

32. File Management Summary
Replica Catalog: Map Logical to Site files
Site A
Site B
Storage Element A
Storage Element B
If files are available from more than one storage site (because they have been transferred using the File Transfer service), we need a service that keeps track of them. This service is the Replica Catalog, it contains the mappings of the Logical File Names to their Site File Names, i.e. the name by which the file can be accessed through the Storage Element in a given site.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

33. File Management Summary
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Site A
Site B
Storage Element A
Storage Element B
Next, we need to provide a service that makes a choice between the available replicas based on a set of information available from the Grid: network bandwith, storage element access speed, etc.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

34. File Management Summary
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Site A
Site B
Storage Element A
Storage Element B
Many file types need some kind of pre- and/or post-processing before and/or after transfer. These steps might be related to data extraction, conversion, encryption for pre-processing and importation, conversion, decryption, etc for post-prosessing. Also, as a Quality-of-Service step the file may be validated after transfer as a post-processing step (by checking its checksum, for example). This service should be customizable by the applications that use certain file types.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

35. File Management Summary
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Storage Element A
Storage Element B
Automated replication based on a subscription model is also desirable especially for HEP, where the data coming from the LHC will have to be distributed automatically over the world. The subscription semantics may be defined through file name patterns, data locations, etc. This service should also be customizable by the Virtual Organizations setting it up.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

36. File Management Summary
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Storage Element A
Storage Element B
There is another desirable automatic replication mechanism that should be available through a service, which is automated replication to balance the access load on ‘popular’ files. This service needs to store and analyze user access patterns on the files available through the Grid. Replication is triggered by rules that are applied to the usage patterns on the existing files.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

37. Replica Manager: ‘atomic’ replication operation single client interface orchestrator
File Management
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Storage Element A
Storage Element B
The user does not want to be burdened to contact each of these services directly, taking care of the correct order of operations and making sure that errors are caught and handled accordingly. This functionality is provided by the Replica Manager which acts as the single interface to all replication services. As an additional QoS functionality it should provide transactional integrity of the steps involved in the high-level usages of the underlying services. It should be able to gracefully recover interrupted transfers, catalog updates, pre- and post-processing steps, etc.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

38. Replica Manager: ‘atomic’ replication operation single client interface orchestrator
File Management
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Metadata: LFN metadata
Transaction information
Access patterns
Storage Element A
Storage Element B
A helper service to be used by the File Management services described until now is the metadata service. It should be able to store all metadata associated with Data Management, like attributes on LFNs, patterns for the load balancing service, transaction locks for the replica manager, etc.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D

39. Security File Management
Replica Manager: ‘atomic’ replication operation single client interface orchestrator
File Management
Replica Catalog: Map Logical to Site files
Replica Selection: Get ‘best’ file
Security
Pre- Post-processing: Prepare files for transfer
Validate files after transfer
Replication Automation: Data Source subscription
Site A
Site B
Load balancing: Replicate based on usage
Metadata: LFN metadata
Transaction information
Access patterns
Storage Element A
Storage Element B
Across all of these services we need to impose the stamp of Security, which means that the services need to authenticate and authorize their clients. In order to provide a higher level of consistency between the replicas, we require the users to transfer all administrative rights on the files that they register through the Replica Manager. In order to administer their files they therefore will need to go through the Replica Manager again – administering the files directly through the Storage will be denied. This enables the Replica Manager to act for the user on the user’s behalf – automated replication would not be possible without this restriction.
File Transfer
File A
File X
File A
File C
File B
File Y
File B
File D