1. EMI INFSO-RI-261611 Catalogue synchronization & ACL propagation Fabrizio Furano (CERN IT-GT)

2. EMI INFSO-RI-261611 The problem The Working Group Goal and milestones Message brokers Communication protocols and guidelines Conclusions 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Outline

3. EMI INFSO-RI-261611 Various catalogues keep information that is related – E.g. LFC keeps info about the content of remote SEs, each one with its catalogue – A change in the permissions of a file in LFC is not automatically reflected by the peripheric catalogue – If a SE looses a file, the LFC des not know Keeping them in sync is a very hard problem 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The problem

4. EMI INFSO-RI-261611 Actually, the updates to the various catalogues are performed as external actions – By the jobs, as long as they do not die/crash in the wrong moment – As scheduled rounds of massive synchronizations 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The problem

5. EMI INFSO-RI-261611 Current consistency model is not resilient to failures – Storage failures lead to dangling entries to be cleaned up manually. Catalogue failures lead to orphaned files. Namespace scanning for diffs is an expensive workaround 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The problem

6. EMI INFSO-RI-261611 Make the various catalogues/SE able to talk to each other – In order to exchange messages that keep them synchronized – 2 ways: LFC->SE to propagate changes in the permissions SE->LFC to propagate info about lost files – These are just starting examples, what can be done can be even much more interesting 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The proposal

7. EMI INFSO-RI-261611 In other words: – Disasters and misalignments will still happen Nobody can prevent a file from disappearing – Let’s design a mechanism that ruggedizes the catalogues by means of notifications – Let’s design a mechanism to propagate the changes in the LFC to all the interested SEs 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The proposal

8. EMI INFSO-RI-261611 The kickoff mail was sent a few days ago in emi-jra1-data Working group: – Jean-Philippe B. – Paul M. – Riccardo Z. – Michele D. – Fabrizio F. (Me) 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The working group

9. EMI INFSO-RI-261611 Proposed demonstrators to use reliable message (i.e. industry standard MQ) as backbone of the reliability – All interested catalogues can “subscribe” for permissions that changed in the LFC – Lost files can be broadcast on the “lost” topic to interested catalogues Somehow possible also for corrupted “bad” files (e.g. not readable) 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Milestone

10. EMI INFSO-RI-261611 Looking for good ways to reliably communicate and cooperate Communication 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation SE1 SEn SE2 SE or exp. catalogue subscribes to the relevant topics (e.g. “Changes”) SE sends to the appropriate topics (e.g. “Lost”) LFC Subscribes to the relevant topics (e.g. “Lost”) SE Sends to the appropriate topics (e.g. “Changes”) LFC ATLAS catalogue ATLAS catalogue Broker(s)

11. EMI INFSO-RI-261611 Industry standard MQ brokers, like ActiveMQ will constitute the backbone Need to: – Agree on the needed messages (content+semantics) – Decide the implementation guidelines 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Message brokers

12. EMI INFSO-RI-261611 Several APIs are available, usable from nearly every language Each API talks to the broker using a protocol. Mostly, in our case: – OpenWire (binary) – STOMP (XML based) The broker speaks them all and assures the interoperability The idea is not to enforce one protocol, just give usage guidelines 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation APIs and protocols

13. EMI INFSO-RI-261611 Although compatible, each API/Protocol has its own idiosyncrasies. E.g.: The recent release of the C++ library is simply broken. I had to do all the tests with (their) trunk, with very good results. The reference STOMP implementation that I tested does not support recovery. Any glitch silently makes a zombie client until manual restart. Probably many others. – Need to sort out the needed features 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation APIs and protocols

14. EMI INFSO-RI-261611 Many aspects have still to be sorted out (E.g. security requirements), but at least we started well. Making catalogues and SEs interact seems a good way to solve the consistency problem The messaging broker tech is not new, at least we know that it works The performance seems good, eventually it should also be sufficiently scalable Next goal: demo for December 2010 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Conclusions

15. EMI INFSO-RI-261611 Thank you 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611