EMI INFSO-RI EMI Data EMI Data, the unified European Data Management Middleware Patrick Fuhrmann (DESY) EMI Data Area lead

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 2 Credits Alejandro Alvarez Alex Sim Claudio Cacciari Christian Bernardt Christian Loeschen Dirk Duellmann Elisabetta Ronchieri Fabrizio Furano Giuseppe Fiameni Giacinto Donvito Giuseppe Lo Presti Jon Kerr Nilsen Jan Schaefer Jean-Philippe Baud Karsten Schwank Michele Carpene Michele Dibenedetto Michail Salichos Mischa Salle Oscar Koeroo Oliver Keeble Paul Millar Ralph Mueller-Pfefferkorn Ricardo Rocha Riccardo Zappi Tigran Mkrtchyan Zsolt Molnar Zsombor Nagy Our wiki :

EMI INFSO-RI The EMI Facts –From 22 Partners to EMI-Data –The EMI-Data shopping cart. –The EMI(-Data) release plan. The EMI-Data Mission –Standardization WebDAV NFS 4.1 / pNFS SSL for SRM –Interoperability ARGUS Client library consolidation GLUE 2.0 UNICORE Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 3 Outline Mission (Cont) –Catalogue Synchronization Conclusion

EMI INFSO-RI EMI Overview - EGI TF, Amsterdam 416/09/2010 Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 4 EMI Factsheet Budget : about 24 Million Euros Funding : about 50% by EU-FP7, rest by partners Covers : JRA, SA and NA Partners : 22 Middlewares: Arc, gLite, UNICORE and dCache

EMI INFSO-RI Why Apr 21, EMI Data,.... EGI UF2011, Vilnius The European Middleware Initiative (EMI)project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve distributed computing middleware for deployment in EGI, PRACE and other distributed e- Infrastructures

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 6 EMI in context EMI EGI, PRACE, WLCG,OSG Requirements Releases SLAs & Support Collaborations DCI collaborations 6 ESFRI, VRCs ESFRI, VRCs StratusLab VENUS-C EDGI IGE SIENA Standards Industry Standards Industry Collaborations Stolen from Alberto Di Meglio

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 7 “User” Feedback process EGI, Heavy User: HEP, Astronomy and Astro Physics, Life and Earth Science Technical Coordination Board Project Technical Board Project Technical Board EMI Technical Director Project Executive Board Project Executive Board EMI Releases WLCG, OSG, ? !! YOU !!

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 8 EMI Data EMI Data in context dCache, StoRM, DPM, FTS, LFC, GFAL, arc- libs, UNICORE- SMS, etc Information system, accounting, bookkeeping ARGUS, VOMS, UNICORE- Gate, gridSite, etc A-REX, UAS- Compute, WMS, CREAM, MPI, etc

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 9 What does EMI-Data provide ?

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 10 The EMI shopping cart Professional Storage Solutions DPM dCache Fits all sizes (IKEA approach) Reliable File Transport Service File Location and meta data Service (LFC) Generic Meta Data Service (AMGA) DOG Happy Dancing Data Access and control library(s)

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 11 Release Plan Start EMI 0 EMI 1 EMI 2 EMI 3 Support & Maintenance Supp. & Maint. Major releases Stolen from Alberto Di Meglio 01/05/201031/10/ /04/ /04/201228/02/2013 Kebnekaise, Lappland, Sw, 2100m Giebnegáisi Kebnekaise, Lappland, Sw, 2100m Giebnegáisi

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 12 The Mission

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 13 The Mission Fixing of issues based on the experience of operating the infrastructures for some years. Improving or creating interoperability between components and middle-wares. Reducing components by merging functionality or removing duplication. Applying standards where available Standardizing EMI-Data mechanisms with “standardization bodies” e.g. OGF EGI : Attracting resp. enabling new communities. Becoming competitive and attractive – Standards – Professional Support – Strict quality monitoring

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 14 Mission overview

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 15 Standardization

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 16 Standardization : WebDAV Very useful for new (non-LHC) communities. ITEF Standard Allows “File system like” access with Mac OS Linux Windows

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 17 Standardization : NFS 4.1 / pNFS Simplicity Regular mount-point and real POSIX I/O Can be used by unmodified applications (e.g. Mathematica..) Data client provided by the OS vendor Smart caching (block caching) development done by OS vendors Performance pNFS : parallel NFS (first version of NFS which support multiple data servers) Clever protocols, e.g. Compound Requests

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 18 Standardization : NFS 4.1 / pNFS NetAppTest DESY Production (OnTap 8.1) NetApp Blue Arc dCache Next Golden Release (1.9.12) DPM 2011 Q1Q2Q3Q StoRM GPFS nativeGPFS pNFS pNFS in official kernel Linux distributions (RH6.2…) pNFS Enabled Kernel Industry Beta read-only DESY Linux pNFS kernel for SL5 There might be issues with 38 Availability for production use Beta Read/Write

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 19 Standardization : NFS 4.1 / pNFS Mar 16, 2011 Göttingen, NFS 4.1/pNFS, Patrick Fuhrmann 19 Patrick Fuhrmann, dCache.org 19 The DESY Grid Lab Operated by Yves Kemp Dmitri Ozerov The DESY Grid Lab Has been enlarged and will start testing end of March’11 Cern NFS 4.1 evaluation setup Contact Andrea Sciaba and Ricardo Rocha DESY Grid Lab available for more than 9 months to evaluate protocols and systems. Publications at CHEP and HEPIX Presentation on NFS 4.1 Details In the afternoon.

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 20 Standards : SRM over SSL Presentation on SRM/SSL Details After these messages

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 21 Spin off standards

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 22 Storage Accounting Record (StAR) If there is no standard for storage accounting yet, do anyone actually need it? Developing a storage infrastructure – We need to know how much storage space is used, by which group/user, on which storage media – To know where to put the money when increasing the storage space – To know who to ask for the money to increase the storage space Basis for billing – Storage is expensive – Some non-academic resource owners may not like to give it away for free Jon prepared a draft definition of a StAR -> OGF existing working group (Jon becoming Co-Chair) Presentation on StAR Wednesday afternoon

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 23 Delegation using SSLv3

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 24 Interoperability

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 25 Interoperability ARGUS integration ARGUS : Authorization system Allows local and centralized configuration (and both) SE’s, LFC and FTS starting with ARGUS blacklisting Client Library consolidation (ARC, gLite) Goal : Merge ARC and gLite client library functionality SRM Client Catalogue access … Activity started in Feb at CERN with F2F. No casualties were reported.

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 26 Interoperability Migration from GLUE 1.3 to GLUE 2.0 Already agreed before EMI Starting with GLUE 1.3 information published with GLUE 2.0 Followed by clients (between EMI-1 and EMI-2) Running both systems in parallel for awhile. Presentation on GLUE 2 in DM Wednesday afternoon

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 27 Interoperability

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 28 Fixing an intended design flaw

EMI INFSO-RI Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 29 Fixing an intended design flaw Presentation on Catalogue Synchronzation Late this afternoon

EMI INFSO-RI AMGA Demo Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 30 Find EMI Data in Vilnius Monday Tuesday Wednesday 14:00 16:00 IN

EMI INFSO-RI EMI Data EMI Data is a good opportunity to get our storage management middleware into a maintainable shape. Standardization is the way to get broader acceptance by other communities, which is especially important for EGI. EMI-Data will become THE competitor in Storage Management in Europe. Everybody can join or may provide suggestions through WLCG or EGI.eu. Apr 21, 2011 EMI Data,.... EGI UF2011, Vilnius 31 Conclusions

EMI INFSO-RI Further reading EMI/EmiJra1T3Data Apr 21, EMI Data,.... EGI UF2011, Vilnius EMI is partially funded by the European Commission under Grant Agreement INFSO-RI