1. Data Management & Information Systems
Markus Schulz – SA3 - CERN
OGF - EGEE-II User Forum
Manchester - 9 May 2007

2. Ask questions! Disclaimer
Material that went into this presentation has been provided by many developers from inside JRA1, SA3, NA4 and external contributors
Many thanks
Ask questions!
OGF - EGEE II User Forum - Manchester - 9 May 2007

3. Data Management LFC SRM gfal and lcg-utils FTS AMGA
DPM
gfal and lcg-utils
FTS
AMGA
Medical data management components
Hydra Keystore, Data Encryption, DPM-DICOM
OGF - EGEE II User Forum - Manchester - 9 May 2007

4. EGEE Data Management VO Frameworks User Tools Data Management
lcg_utils
FTS
Cataloging
Storage
Data transfer
GFAL
Vendor
Specific
APIs
(RLS)
LFC
SRM
(Classic
SE)
gridftp
RFIO
OGF - EGEE II User Forum - Manchester - 9 May 2007

5. LFC LCG File Catalog LHC Computing Grid File Catalog
Large Hadron Collider Computing Grid File Catalog

6. LCG “File” Catalog The LFC stores mappings between
Users’ file names
File locations on the Grid
The LFC is accessible via
CLI, C API, Python interface, Perl interface
Supports sessions and bulk operations
Data Location Interface (DLI)
Web Service used for match making:
given a GUID, returns physical file location
ORACLE backend for high performance applications
Read-only replication support …
File replica 2
GUID
File replica 1
File replica m
LFC file name 1
LFC file name n
OGF - EGEE II User Forum - Manchester - 9 May 2007

7. LFC features Hierarchical Namespace GSI security
Permissions and ownership
ACLs (based on VOMS)
Virtual ids
Each user is mapped to (uid, gid)
VOMS support
To each VOMS group/role corresponds a virtual gid
/grid
/vo
/data
LFC
DLI
lfc-ls –l /grid/vo/
file
lfc-getacl /grid/vo/data
OGF - EGEE II User Forum - Manchester - 9 May 2007

8. Secondary groups support
What's new ?
LFC bulk operations
New method: lfc_getreplicas
Greatly improves replicas listing performance
Secondary groups support
Since LFC version (in production)
OGF - EGEE II User Forum - Manchester - 9 May 2007

9. User 2 from VO1 with VOMS Role
Before LFC version 1.6.3
1. Creates directory
LFC dir 1
775 (uid1, gid1)
User 1 from VO1
Mapped to (uid1, gid1)
2. Tries to create file
Before secondary groups
User 2 could not register a file in dir1
As (s)he did not belong to gid1
Solution: set ACLs on dir1
NEW
User 2 from VO1 with VOMS Role
Mapped to (uid2, gid2)
OGF - EGEE II User Forum - Manchester - 9 May 2007

10. User 2 from VO1 with VOMS Role
Since LFC version 1.6.3
1. Creates directory
LFC dir 1
775 (uid1, gid1)
User 1 from VO 1
Mapped to (uid1, gid1)
2. Tries to create file
With secondary groups
User 2 can register a file in dir1
As (s)he belongs to gid2 and gid1
But: User 1 cannot register a file in a directory created by User 2, if (s)he does not have the same VOMS Role !
NEW
User 2 from VO1 with VOMS Role
Mapped to (uid2, gid2, gid1)
Also belongs to VO1
OGF - EGEE II User Forum - Manchester - 9 May 2007

11. DPM (Disk Pool Manager)

12. Storage Element Storage Resource Manager (SRM)
hides the storage system implementation (disk or active tape)
handles authorization
translates SURLs (Storage URL) to TURLs (Transfer URLs)
disk-based: DPM, dCache,+; tape-based: Castor, dCache
File I/O: posix-like access from local nodes or the grid
GFAL (Grid File Access Layer)
OGF - EGEE II User Forum - Manchester - 9 May 2007

13. What is a DPM ? Disk Pool Manager Manages storage on disk servers
SRM support
1.1
2.1 (for backward compatibility)
2.2 (released in DPM version 1.6.3)
GSI security
ACLs
VOMS support
Secondary groups support (see LFC)
OGF - EGEE II User Forum - Manchester - 9 May 2007

14. Target: small to medium sites
DPM strengths
Easy to use
Hierarchical namespace
$ dpns-ls /dpm/cern.ch/home/vo/data
Easy to administrate
Easy to install and configure
Low maintenance effort
Easy to add/drain/remove disk servers
Target: small to medium sites
Single disks --> several disk servers
OGF - EGEE II User Forum - Manchester - 9 May 2007

15. DPM: user's point of view
/domain
/home
CLI, C API,
SRM-enabled client, etc.
/vo
(uid, gid1, …)
DPM
head node
file
data transfer
DPM Name Server
Namespace
Authorization
Physical files location
Disk Servers
Physical files
Direct data transfer from/to disk server (no bottleneck)
External transfers via gridFTP
DPM
disk servers
OGF - EGEE II User Forum - Manchester - 9 May 2007

16. GFAL & lcg_util Data management access libs. GFAL
Shield users from complexity
Interacts with information system, catalogue and SRM-SEs
GFAL
Posix like C API for file access
SRMv2.2 support
User space tokens correspond to
A certain retention policy (custodial/replica)
A certain access latency (online/nearline)
lcg_util (command line + C API )
Replication, catalogue interaction etc.
OGF - EGEE II User Forum - Manchester - 9 May 2007

17. LFC & DPM deployment status
EGEE Catalog
110 LFCs in production
37 central LFCs
73 local LFCs
EGEE SRM Storage Elements
CASTOR
dCache
DPM
96 DPMs in production
Supporting 135 VOs
LFC and DPM
Stable and reliable production quality services
Well established services
Require low support effort from administrators and developers
Storage Element instances
published in EGEE’s Top BDII
OGF - EGEE II User Forum - Manchester - 9 May 2007

18. FTS overview
gLite File Transfer Service is a reliable data movement fabric service (batch for file transfers)
FTS performs bulk file transfers between multiple sites
Transfers are made between any SRM-compliant storage elements (both SRM 1.1 and 2.2 supported)
It is a multi-VO service, used to balance usage of site resources according to the SLAs agreed between a site and the VOs it supports
VOMS aware
OGF - EGEE II User Forum - Manchester - 9 May 2007 18

19. FTS
Why is it needed ?
For the user, the service it provides is the reliable point to point movement of Storage URLs (SURLs) and ensures you get your share of the sites’ resources
For the site manager, it provides a reliable and manageable way of serving file movement requests from their VOs and an easy way to discover problems with the overall service delivered to the users
For the VO production manager, it provides ability to control requests coming from his users
Re-ordering, prioritization,…
The focus is on the “service” delivered to the user
It makes it easy to do these things well with minimal manpower
OGF - EGEE II User Forum - Manchester - 9 May 2007 19

20. FTS: key points Reliability Security Service and performance
It handles the retries in case of storage / network failures
VO customizable retry logic
Service designed for high-availability deployment
Security
All data is transferred securely using delegated credentials with SRM / gridFTP
Service audits all user / admin operations
Service and performance
Service stability: it is designed to efficiently use the available storage and network resources without overloading them
Service recovery: integration of monitoring to detect service-level degradation
OGF - EGEE II User Forum - Manchester - 9 May 2007 20

21. Service scale
Designed to scale up to the transfer needs of very data intensive applications
Currently deployed in production at CERN
Running the production WLCG tier-0 data export
Target rate is ~1 Gbyte/sec 24/7
Over 9 petabytes transferred in last 6 months > 10 million files
Also deployed at ~10 tier-1 sites running a mesh of transfers across WLCG
Inter-tier1 and tier-1 to tier-2 transfers
Each tier-1 has transferred around 0.2 – 0.5 petabytes of data
OGF - EGEE II User Forum - Manchester - 9 May 2007 21

22. The gLite AMGA Metadata Catalogue

23. Metadata in EGEE Metadata is information about data stored in files
usually lives in relational databases
AMGA is a joint JRA1-NA4 development
Used by several application domains ( BioMed, HEP, EarthObs….)
Implementation:
SOAP and Text front-ends
Streamed Bulk Operations ----> performance
Supports single calls, sessions & connections
SSL security with grid certs (X509)
and others, passwords, Kerberos
Own User & Group management + VOMS
PostgreSQL, Oracle, MySQL, SQLite backends
Query parser supports good fraction of SQL:
Access permissions per directory/entry via ACLs
AMGA integrates support for replication of metadata
Asynchronous replication: Ideal for WAN

24. AMGA Clients & APIs AMGA Clients (for setup, administration)
Shell-like client
Graphical Browser (Python)
Many Programming APIs
Diverse user community requested/provided C/C++, Java, Python, Perl, PHP
SOAP interface
Works with gSOAP, Axis, PySOAP

25. Performance Performance comparable to direct DB access
C++, TCP streaming protocol, very fast SSL sessions
Logarithmic Scale!
100
1000
10000
100000
1e+06 1 10
Throughput [entries/s]
# clients
AMGA 1000 rows
JDBC 1000 rows
AMGA 1 row
JDBC 1 row
Throughput comparison between AMGA and direct access via JDBC reading same table on a LAN

26. Scale LHCb (HEP VO use case) 100 Million entries successfully tested!
150GB data
entries/day insert rate expected
10 entries/second read-rate
Uses ORACLE RAC backend
For most demanding use cases

27. Conclusions AMGA provides Grid Layer to relational databases:
Abstraction of different DB vendors
Efficient LAN/WAN access
Fast X509 Grid security, VOMS integration
Rich set of features: Transactions, Views, Sequences, complex Joins....
AMGA is building block for distributed databases:
Asynchronous replication
Management tools for distributed AMGA servers
BUT: AMGA misses distributed Query Processing
AMGA in production replacing conventional RDBMS based systems:
Very large, high performance installation for HEP
High security solutions for MDM

28. Encrypted Data Storage

29. Motivation Medical community as the principal user
large amount of images
privacy concerns vs. processing needs
ease of use (image production and application)
Strong security requirements
anonymity (patient data is separate)
fine grained access control (only selected individuals)
privacy (even storage administrator cannot read)
Legacy service in use, based on gLite-1.5
Described components are under development

30. Building Blocks Hospitals: Grid: SE = SRM + gridftp + I/O
DICOM = Digital Image and COmmunication in Medicine
Grid: SE = SRM + gridftp + I/O
and a client (application processing an image)
Goal: data access at any location
DICOM SE
SRM
gridftp
I/O

31. Exporting Images “wrapping” DICOM :
anonymity: patient data is separated and stored in AMGA
access control: ACL information on individual files in SE (DPM)
privacy: per-file keys
distributed among several Hydra key servers
fine grained access control
Image is retrieved from DICOM and processed to be “exported” to the grid.
AMGA metadata
Hydra KeyStore
Hydra KeyStore
Hydra KeyStore
gridftp
patient data
keys
trigger
SRMv2
image
file ACL
I/O
DICOM
DICOM-SE

32. Accessing Images image ID is located by AMGA
key is retrieved from the Hydra key servers
file is accessed by SRM (access control in DPM)
data is read and decrypted block-by-block in memory only (GFAL and hydra-cli)---> useful for all
Still to be solved:
ACL synchronization among SEs
Hydra KeyStore
Hydra KeyStore
Hydra KeyStore
AMGA metadata
gridftp
2. keys
1. patient look-up
SRMv2
3. get TURL
GFAL
image
I/O
4. read
DICOM
DICOM-SE

33. Information Systems R-GMA BDII (ldap based information system)
OGF - EGEE II User Forum - Manchester - 9 May 2007

34. Relational Grid Monitoring Architecture R-GMA
Publish Tuples
Producer
application
Producer
Service
API
SQL “INSERT”
Register
Registry
Service
Query
Tuples
SQL “SELECT”
Locate
Send Query
Consumer
application
Consumer
Service
API
Receive Tuples
Schema
Service
SQL “CREATE TABLE”
For users R-GMA appears similar to a single relational database.
Implementation of GGF’s Grid Monitoring Architecture (GMA)
Rich set of APIs (WebBrowsers, Java, C/C++, Python)
Backbone of EGEE monitoring (almost every activity leaves traces)
See Dashboard, Realtime Monitor about 20 tools
Used by EGEE accounting as transport
OGF - EGEE II User Forum - Manchester - 9 May 2007

35. Service discovery SD provides simple methods for locating services
hides underlying information system (simplified use)
plug-ins for R-GMA, BDII and XML files
API available for Java, C/C++ and command line tools
OGF - EGEE II User Forum - Manchester - 9 May 2007

36. The Information System
Berkeley Data Base Information Index
BDII
top-level
FCR
Queries
(15HZ)
WMS
VO specific filter, based on live status
2 minutes WN
Site
BDII
site-level UI
FTS
Based on ldap
Standardized information provider (GIP)
GLUE-1.3 schema
Used with 230+ sites
Roughly 60 instances in EGEE
Top level BDII at CERN 15HZ query rate
>20MByte of data
BDII
resource
MDS
GRIS
provider
provider
OGF - EGEE II User Forum - Manchester - 9 May 2007

37. Grid foundation: Information Systems
Generic Information
Provider (GIP)
Provides LDIF
information about
a grid service in
accordance to the
GLUE Schema
BDII: Information system in gLite 3.0 (by LCG)
LDAP database that is
updated by a process
More than one DBs is used
separate read and write
A port forwarder is used internally
to select the correct DB
Freedom of choice portal: VOs can white- or black-list resources so that BDII DBs are updated accordingly
Sites failing Site Functional Tests may also be excluded
GIP
Cache
Provider
Plugin
LDIF
File
Config
File
2171
LDAP
2172
2173
2170
Port Fwd
Update DB
&
Modify DB
Swap DBs
OGF - EGEE II User Forum - Manchester - 9 May 2007

38. Inside A BDII FCR Write to cache 2171 LDAP 2172 LDAP 2173 LDAP
Update DB
&
Modify DB
Write to cache
Write to cache
Write to cache
ldapsearch
Swap DBs
2170
Port Fwd
2170
Port Fwd
OGF - EGEE II User Forum - Manchester - 9 May 2007

39. Load Balanced BDII DNS Round Robin Alias Queries BDII 2170 BDII 2170
OGF - EGEE II User Forum - Manchester - 9 May 2007

40. Queries Analysis of a log file from one top-level BDII over 4 hours
Multiply by 8 to get the values for the load balance service (15Hz)
No of Queries
Query
6075
Find the Close CE to an SE
5475
Find the VOs SA for an SE
5043
Find all SRMs
4791
Find an SE
2432
Find the Close SE to a CE
2117
Find all Services for a VO
664
Find all CEs for a VO
638
Find all SAs for a VO
479
Find all SubClusters
448
Find the GlueVOView for a CE
OGF - EGEE II User Forum - Manchester - 9 May 2007

41. Queries To a Top-Level BDII
OGF - EGEE II User Forum - Manchester - 9 May 2007

42. Next Steps Short term Medium term
Put the site-level BDIIs on a stand alone node to improve scalability
Run the CE information provider on the site-level BDII
Introduce regional top-level BDIIs
Top spread the query load
Must ensure all have the same quality of service
Medium term
Improve the efficiencies of the queries
Add some form of caching in the existing tools
Improve the query performance of the BDII service
OGF - EGEE II User Forum - Manchester - 9 May 2007

43. Generic Information Provider
GIN BDII
GIN
BDII
ARC
BDII
Used by the GIN group
Generic Information Provider
Provider EGEE
Provider OSG
Provider NDGF
Provider Naregi
Provider Teragrid
Provider Pragma
EGEE
Site
OSG
Site
NDGF
Site
Naregi
Grid
Teragrid
Grid
Pragma
Grid
OGF - EGEE II User Forum - Manchester - 9 May 2007

44. Information Systems Current problems
SLAPD demons on loaded systems can starve
CEs drop out of the system
Move info provider from CE
SiteBDIIs co-hosted on busy systems time out
Loss of an entire site in the info system
Move on large sites to low load node
Improve reliability by fail back top level BDIIs
Needs work in the clients
Scalability tests indicate limits (1-2 years time)
Cache static data more aggressively
Smarter schema (OGF- GLUE)
Change underlying technology
Simple insulation API needed > Standardization (OGF SAGA?)
OGF - EGEE II User Forum - Manchester - 9 May 2007