1. EMI INFSO-RI-261611 EMI Data EMI Data, the unified European Data Management Middleware Patrick Fuhrmann (DESY) EMI Data Area lead (on behalf of many people and slides stolen from all over the place)

2. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 2 Credits Alejandro Alvarez Alex Sim Claudio Cacciari Christian Bernardt Christian Loeschen Elisabetta Ronchieri Fabrizio Furano Giuseppe Fiameni Giacinto Donvito Giuseppe Lo Presti Jon Kerr Nilsen Jan Schaefer Jean-Philippe Baud Dmitry Ozerov Yves Kemp Karsten Schwank Michele Carpene Michele Dibenedetto Michail Salichos Mischa Salle Oscar Koeroo Oliver Keeble Paul Millar Ralph Mueller-Pfefferkorn Ricardo Rocha Riccardo Zappi Tigran Mkrtchyan Zsolt Molnar Zsombor Nagy Our wiki : https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T3Data

3. EMI INFSO-RI-261611 EMI, the facts EMI-Data, the components EMI-Data, the mission EMI-Data, selected Topics –Topic 1 : Interoperability and reduction of components Client library consolidation –Topic 2 : Fixing flaws and evolution Catalogue and SE synchronization –Topic 3 : Standardization Example 1 : WebDAV Example 2 : NFSv4.1/pNFS July 6, 2011 EMI Data, superB, Ferrara, IT 3 Outline

4. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 4 EMI, the project EMI, the facts

5. EMI INFSO-RI-261611 EMI Overview - EGI TF, Amsterdam 516/09/2010 July 6, 2011 EMI Data, superB, Ferrara, IT 5 EMI Factsheet

6. EMI INFSO-RI-261611 EMI Overview - EGI TF, Amsterdam 616/09/2010 July 6, 2011 EMI Data, superB, Ferrara, IT 6 EMI Factsheet Budget : about 24 Million Euros Funding : about 50% by EU-FP7, rest by partners Covers : JRA, SA and NA Partners : 22 Middlewares: Arc, gLite, UNICORE and dCache

7. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 7 For more facts For more facts on EMI in general, see Francesco’s presentation from this morning on EMI/EGI Issues

8. EMI INFSO-RI-261611 Director’s Mantra July 6, 20118 EMI Data, superB, Ferrara, IT The European Middleware Initiative (EMI) project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve distributed computing middleware for deployment in EGI, PRACE and other distributed e- Infrastructures

9. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 9 Translates to EMI data as Maintain and improve the currently existing production WLCG software stack ! Allow new communities to utilize the EMI middle-wares by applying standards

10. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 10 Components EMI, the components

11. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 11 The EMI Pie MPI A-REX UAS-Compute WMS gridSite ARGUS VOMS UNICORE-Gate accounting Information system bookkeeping dCache StoRM UNICORE-SMS FTS, LFC, DPM, GFAL Arc-libs 63 components and about 350 packages

12. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 12 The EMI Pie dCache StoRM UNICORE-SMS FTS, LFC, DPM, GFAL Arc-libs 63 components and about 350 packages

13. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 13 What does EMI-Data provide ?

14. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 14 The EMI shopping cart Reliable File Transport Service File Location and meta data Service (LFC) Professional Storage Solutions DPM dCache Fits all size

15. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 15 EMI, the storage elements SRM Disk Storage Layer HSM Interface HSM dCache DPM Any Other GPFS TSM Any FS ENSTORE

16. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 16 EMI, the storage elements SRM Disk Storage Layer HSM Interface HSM dCache DPM Any Other GPFS TSM Any FS ENSTORE !!! Something for everyone

17. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 17 The Mission EMI Data mission

18. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 18 The Mission Fixing of issues based on the experience of operating the infrastructures for some years. Improving or creating interoperability between components and middle-wares. Reducing components by merging functionality or removing duplication. Applying standards where available Standardizing EMI-Data mechanisms with “standardization bodies” e.g. OGF EGI : Attracting resp. enabling new communities. Becoming competitive and attractive by : – Standards – Professional Support – Strict quality monitoring

19. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 19 Some more examples Defining (with OGF) and implementing an Storage Accounting Record Migrating the security of the Storage Resource Manager protocol from GSI (httpg) to standard SSL/X509. Migrating to next version of the information provider schema GLUE2.0 Improving the File Transfer Service by integrating the load of the network and the storage element backend. For the entire list, have a look at : – https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T3Data

20. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 20 Selected Topics EMI, some selected topics

21. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 21 Topic 1 Client library consolidation

22. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 22 Component consolidation dCache ARCDataLib gLiteDataLib DataInterface

23. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 23 Component consolidation dCache EMIDataLibEMIjDataLib DataInterface

24. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 24 Topic 2 Catalogue / Storage element synchronization

25. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 25 Catalogue and Storage Elements Storage Elements are storing the actual data locally. Catalogues map logical filenames to actually locations URLs. Typical store operation : – Store file in SE – Add entry (URL) to the catalogue under the logical file name May possibilities for SE’s and catalogues to get out of sync as the store and ‘add entry’ operations are not transactional in WLCG. This results in – Dark data (files in SE but not in catalogue) – Dangling references (Pointer in catalogue, pointing into the void) Over time, catalogues and SE’s diverge more and more. Problem

26. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 26 How to solve this issue Two different approaches

27. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 27 Two possible solutions Synchronizing catalogues and SE’s using message passing SE or Catalogue specific plug-in Generic Adapter Messaging infrastructure Generic Adapter DPM, StoRM or dCache LFC or experiment catalogue Talk to Fabrizio on details.

28. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 28 Already in EMI-1 Command Line Interface Messaging infrastructure Generic Adapter LFC or experiment catalogue List of removed files Manual interaction

29. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 29 Second approach (in discussion) Dynamic location catalogues The catalogue, on request, builds a list of URLs for logical filenames. The URLs automatically invalidate after some time (hours). Dynamic catalogue Storage elements Standard Clients Request for logical filename Cache miss Fill cache with query results Return up-to-date information

30. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 30 Example on how EMI works Synchronizing catalogues and SE’s guaranties continuity of production environment. Dynamic catalogues is part of the data management evolution.

31. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 31 Selected topics Topic 3 Standards WebDAV NFS 4.1 / pNFS

32. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 32 Standardization

33. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 33 Selected Topics Example 1 WebDAV

34. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 34 Standardization : WebDAV Very useful for new (non-LHC) communities. ITEF Standard Allows “File system like” access with Mac OS Linux Windows

35. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 35 Standardization : WebDAV Logical Filename  Replica at site A  Replica at site B StandardBrowser

36. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 36 Selected Topics Example 2 NFS v4.1 / pNFS

37. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 37 What’s NFSv4.1/pNFS ? CITI, at the University of Michigan, is funded by major storage providers to coordinate the pNFS effort and provide reference implementations. Group meets three times a year to check interoperability.

38. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 38 What is NFS4.1/pNFS and how does it work ? Stolen from : http://www.pnfs.com/

39. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 39 Why would one need it ? Benefits of Parallel I/O Delivers Very High Application Performance Allows for Massive Scalability without diminished performance Benefits of NFS (or most any standard) Ensures Interoperability among vendor solutions Allows Choice of best-of-breed products Eliminates Risks of deploying proprietary technology Stolen from : http://www.pnfs.com/

40. EMI INFSO-RI-261611 July 6, 2011 Why would we need it ? Simplicity Regular mount-point and real POSIX I/O Can be used by unmodified applications (e.g. Mathematica..) Data client provided by the OS vendor Smart caching (block caching) development done by OS vendors Performance pNFS : parallel NFS (first version of NFS which support multiple data servers) Clever protocols, e.g. Compound Requests 40 EMI Data, superB, Ferrara, IT

41. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 41 NFS v4.1 / pNFS availability EMI server o dCache : production version with EMI 1 o DPM : prototype, ready for EMI-2 Linux Kernel o Completed in 2.6.39 o Back-port of pNFS into RH 6.2 Industry o NetApp OnTab 8.1 o Blue Arc in August o Other vendors : code ready but not officially available

42. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 42 Testing and deployment NFS 4.1 / pNFS tested for more than 9 month at the DESY GridLab (50% Tier II). Stable and performing well. First production dCache NFS 4.1 / pNFS system at DESY for photon science. Other WLCG sites including Tier I’s start with testing NFS 4.1 / pNFS.

43. EMI INFSO-RI-261611 July 6, 2011 EMI Data, superB, Ferrara, IT 43 Competition NFS 4.1 / pNFS is a great opportunity for Open Source Projects (EMI) to compete with industry and of course the other way around.

44. EMI INFSO-RI-261611 EMI Data EMI Data is a good opportunity to get our storage management middleware into a maintainable shape and at the same time provides a platform for evolution. Standardization is the way to get broader acceptance by other communities, which is especially important for EGI. EMI-Data will become THE competitor in Storage Management in Europe. Everybody can join or may provide suggestions through EGI.eu. July 6, 2011 EMI Data, superB, Ferrara, IT 44 Conclusions

45. EMI INFSO-RI-261611 Further reading https://twiki.cern.ch/twiki/bin/view/ EMI/EmiJra1T3Data July 6, 201145 EMI Data, superB, Ferrara, IT EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611