1. Dynamic Storage Federation based on open protocols
EGI Community Forum 2013
Dynamic Storage Federation
based on open protocols
Adrien Devresse (presenter)
Fabrizio Furano
Patrick Fuhrmann
Paul Millar
Daniel Becker
Oliver Keeble
Ricardo Brito da Rocha
Alejandro Alvarez
Credits to ShuTing Liao (ASGC)

2. Motivations for Federations (1)
Current situation:
→ Data are in groups of distributed of Storage Systems at differents places
→ The location of data is managed by
→ Experiment framework
→ Meta-data catalogs
→ Jobs are ( supposed to be ) placed close to the data
Dynamic Storage Federation- 2

3. Motivations for Federations (2)
Can be improved !
→ What happens to your workflow if 
→ A storage system is offline ?
→ A file is missing ?
→ The meta-data catalog is overloaded ?
→ Listing and/or browsing are expensive/slow operations
→ The file access pattern is clearly not optimal
Dynamic Storage Federation- 3

4. Dynamic Federation Goal ?
Simple Idea :
Federate on the fly 
Grid Distributed Storage,
Cloud Storage,
Any Existing meta-data Catalog,
In a unique namespace
Dynamic Storage Federation- 4

5. Basic Idea of the federation
This is what
we want to see as users
Sites remain independent and participate to a global view
All the metadata interactions are hidden and done on the fly
Aggregation
/dir1
/dir1/file1
/dir1/file2
/dir1/
With 2 replicas
/dir1/file1
/dir1/file2
Storage/MD
endpoint 1
/dir1/file2
/dir1/file3
Storage/MD
endpoint 2
EMI INFSO-RI
Dynamic Storage Federation- 5

6. Open a lot of new possibilities !
→ Reliability, failover
→ Detect offline storage element
→ Detect network problems
→ World wide replica discovery
→ Smart Redirection 
→ Geographical redirection
→ Network optimizations
→ Transparent caching
Dynamic Storage Federation- 6

7. →Transparent data migration → Scalability → Aggressive caching
New possibilities (2)
→ Easy data Migration
→Transparent data migration
→Merge Grid and Cloud Storage namespace
→ Scalability
→ Aggressive caching
→ Protection of your name servers
→ Allow horizontal deployement
→ Load balancing
Dynamic Storage Federation- 7

8. → Multi-stream download ( metalink )
New possibilities (3)
→ Performance
→ Multi-stream download ( metalink )
→ Reduce latency ( GeoIP, local Fed )
→ Low response time ( in memory )
→ Flexibility
→ Algorithmic filename translation
→ Merge any type of Storage ( Grid, Cloud )
→ Use / integrate your own catalog
→ Allow local federations
Dynamic Storage Federation- 8

10. How is it working ? → No Database
→ Fetch the meta-data on demand from the Storage endpoints ( SE )
→ Aggregate the meta-data on the fly
→ Distributed System
Dynamic Storage Federation- 10

11. Scalable Architecture
Dynamic Federation Architecture
Front-end ( Dmlite + Webdav )
→ In Memory Only
→ several layer of cache
→ No Database
→ Meta-data fetched dynamically from the Storage Elements.
Ugr_connector.so ( Entry point )
Descision process
plugin 1
plugin 2
1st layer cache ( RAM process )
2nd layer cache ( Memcached )
Plugin layer
Dav plugin
LFC plugin
S3 plugin
Http plugin
Storage Elements
Dynamic Storage Federation- 11

12. Deployement possibilities
Global federation
Dynamic federation 1
Dynamic federation 3
LHCb
LFC
Dynamic federation 2
Meta-data
catalog
GPFS
EOS
Dynamic Storage Federation- 12

13. Performances test Two storage endpoints: DESY and CERN (poor VM)
One UGR federator at DESY, clients at CERN
10K files are interleaved in a 4-levels deep directory
Oddly-numbered files are at CERN
Evenly-numbered files are at Desy
The test (written in C++) invokes Stat only once per file, using many parallel clients doing stat() at the maximum pace from 3 machines
Dynamic Storage Federation- 13

14. Performances results
Dynamic Storage Federation- 14

15. Dynamic Storage Federation Status
Currently available!
Technically TODAY we can dynamically aggregate:
dCache DAV/HTTP instances
DPM DAV/HTTP instances
LFC DAV/HTTP and old Cns API instances
Cloud DAV/HTTP services
Anything that can be plugged into DMLite (the new architecture for DPM/LFC)
Can be extended to other metadata sources
Dynamic Storage Federation- 15

16. Dynamic Storage Federation Status
→ The system also can load a “Geo” plugin
→ Gives a geographical location to replicas and clients
→ Allows the core to choose the replica
that is closer to the client
→ The one that’s available uses GeoIP (free)
Dynamic Storage Federation- 16

17. We have a stable demo testbed, using HTTP/DAV
Dynamic Storage Federation Demo
We have a stable demo testbed, using HTTP/DAV
Dynamic Storage Federation- 17

18. Dynamic Storage Federation Demo
It is actually 2 demos in one
An ATLAS demo, federating 8 sites, plus LFC as name translator
Note that this is not the full ATLAS repo, it’s just 8 sites.
DESY, KIT, SARA, WUPPERTAL, NDGF, Muenchen, Prague, ASGC
A fully dynamic catalogue-free demo with the EMI testbed
Federating three endpoints.
a DPM instance at CERN
a dCache instance in DESY
one endpoint in LBNL
Dynamic Storage Federation- 18

19. Dynamic Storage Federation News
Very stable, installable from the wiki
Recent improvement: in the case we federate catalogues, the replicas they give can be checked on the fly
Use the catalogue as name translator
Use the catalogue as source of file listings
Check the replicas in the moment they are requested
Next item: ATLAS and Rucio
We have a nice testbed, federating many ATLAS SEs
We want to federate the Rucio services and the LFC(s) seamlessly together
Dynamic Storage Federation- 19

20. Dynamic Storage Federation Links
Documentation and source code
Description of the demo
ederations.htm
Power users wanted
Helping in getting the best out of the system. Your cooperation is very appreciated.
Dynamic Storage Federation- 20

21. Questions ?
Dynamic Storage Federation- 21