EMI INFSO-RI-261611 Catalogue synchronization & ACL propagation Fabrizio Furano (CERN IT-GT)

Slides:



Advertisements
Similar presentations
Support for Fault Tolerance (Dynamic Process Control) Rich Graham Oak Ridge National Laboratory.
Advertisements

FIPA Interaction Protocol. Request Interaction Protocol Summary –Request Interaction Protocol allows one agent to request another to perform some action.
Applicability of Instant Messaging in the Military Command and Control Systems Author: Juha Vermaja Superviser: Jorma Jormakka Instructor: Marko Luoma,
Interpret Application Specifications
Understanding Android Security Yinshu Wu William Enck, Machigar Ongtang, and PatrickMcDaniel Pennsylvania State University.
26-28 th April 2004BioXHIT Kick-off Meeting: WP 5.2Slide 1 WorkPackage 5.2: Implementation of Data management and Project Tracking in Structure Solution.
1 The Google File System Reporter: You-Wei Zhang.
EMI INFSO-RI European Middleware Initiative (EMI) Standardization and Interoperability Florida Estrella (CERN) Deputy Project Director.
CTRP User Call April 3, 2013 Gene Kraus CTRP Program Director.
CH2 System models.
EMI is partially funded by the European Commission under Grant Agreement RI Post EMI Plans and MeDIA Alberto DI MEGLIO, CERN Project Director WLCG.
EMI INFSO-RI SA2 - Quality Assurance Alberto Aimar (CERN) SA2 Leader EMI First EC Review 22 June 2011, Brussels.
EGEE is a project funded by the European Union under contract IST Testing processes Leanne Guy Testing activity manager JRA1 All hands meeting,
A NAMED DATA NETWORKING FLEXIBLE FRAMEWORK FOR MANAGEMENT COMMUNICATION Authors: Daneil Corjuo and Rui L. Aguiar Ivan Vidal and Jamie Garcia-Reinoso Presented.
Second SEED Workshop SEED Phase 2. Plan of Activities.
Reliable Communication in the Presence of Failures Based on the paper by: Kenneth Birman and Thomas A. Joseph Cesar Talledo COEN 317 Fall 05.
This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under.
E-Infrastructure & Data Services Features and implementations 1.
INFNGrid Constanza Project: Status Report A.Domenici, F.Donno, L.Iannone, G.Pucciani, H.Stockinger CNAF, 6 December 2004 WP3-WP5 FIRB meeting.
Overview of user client usage: ARC Iván Márton Zsombor Nagy.
Using the EMI testbed ARC middleware Marek Kočan University of P. J. Šafárik, Košice.
David Adams ATLAS DIAL/ADA JDL and catalogs David Adams BNL December 4, 2003 ATLAS software workshop Production session CERN.
6 th dCache WS | Daniel Becker| 18 April 2012 | 1 Daniel Becker 6 th dCache workshop, Zeuthen, April 18, 2012 The HTTP Federation.
Summary of Data Management Jamboree Ian Bird WLCG Workshop Imperial College 7 th July 2010.
Security Token Service (STS) Design & Development Plans Henri Mikkonen / HIP 3 rd EMI All-Hands Meeting , Padova, Italy.
EMI is partially funded by the European Commission under Grant Agreement RI Discovering Infrastructures with EMI Registry (EMIR) Emidio Giorgio.
UNIVERSITY OF HELSINKI HELSINKI UNIVERSITY OF TECHNOLOGY Semantic Computing Research Group Web Services Service Publishing.
EMI is partially funded by the European Commission under Grant Agreement RI SA2 – Development Tools Andres Abad Rodriguez SA2.4 Tools Activity Leader.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks MSG - A messaging system for efficient and.
EMI INFSO-RI Overview of the EMI development objectives Balázs Kónya (Lund University) EMI Technical Director EMI All Hands Prague, 23rd November.
Objective What is RFT ? How does it work Architecture of RFT RFT and OGSA Issues Demo Questions.
Storage Accounting John Gordon, STFC GDB March 2013.
Information-Centric Networks10b-1 Week 10 / Paper 2 Hermes: a distributed event-based middleware architecture –P.R. Pietzuch, J.M. Bacon –ICDCS 2002 Workshops.
EMI INFSO-RI Accounting John Gordon (STFC) APEL PT Leader.
EMI INFSO-RI EMI Roadmap to Standardization and DCI Collaborations Alberto Di Meglio (CERN) Project Director.
Information-Centric Networks Section # 10.2: Publish/Subscribe Instructor: George Xylomenos Department: Informatics.
EMI INFSO-RI ARC tools for revision and nightly functional tests Jozef Cernak, Marek Kocan, Eva Cernakova (P. J. Safarik University in Kosice, Kosice,
European Middleware Initiative (EMI) The Software Engineering Model Alberto Di Meglio (CERN) Interim Project Director.
The new FTS – proposal FTS status. EMI INFSO-RI /05/ FTS /05/ /05/ Bugs fixed – Support an SE publishing more than.
Distributed Data Management Miguel Branco 1 DQ2 discussion on future features BNL workshop October 4, 2007.
EMI INFSO-RI Technical Overview Balázs Kónya (Lund University) Technical Director 1 st EMI Periodic Review Brussels, 22 June 2011.
CERN IT Department CH-1211 Geneva 23 Switzerland GT WG on Storage Federations Sept 2012 Usages and Goals Summary Fabrizio Furano on behalf.
EMI INFSO-RI Metrics report visualization and reporting Lars Barlindhaug (CERN)
EMI INFSO-RI European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director.
By Nitin Bahadur Gokul Nadathur Department of Computer Sciences University of Wisconsin-Madison Spring 2000.
Distributed Data Access Control Mechanisms and the SRM Peter Kunszt Manager Swiss Grid Initiative Swiss National Supercomputing Centre CSCS GGF Grid Data.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Data Management cluster summary David Smith JRA1 All Hands meeting, Catania, 7 March.
Mutual Exclusion Algorithms. Topics r Defining mutual exclusion r A centralized approach r A distributed approach r An approach assuming an organization.
EMI is partially funded by the European Commission under Grant Agreement RI EMI Registry (EMIR) Shiraz Memon, Ivan Marton, Gabor Szigeti, Laurence.
EMI is partially funded by the European Commission under Grant Agreement RI Common Authentication Library Daniel Kouril, for the CaNL PT EGI CF.
EMI INFSO-RI Catalogue synchronization & ACL propagation Fabrizio Furano (CERN IT-GT-DMS)
EMI INFSO-RI Catalogue synchronization & ACL propagation Fabrizio Furano (CERN IT-GT)
EMI is partially funded by the European Commission under Grant Agreement RI Future Proof Storage with DPM Oliver Keeble (on behalf of the CERN IT-GT-DMS.
Powerpoint Templates Data Communication Muhammad Waseem Iqbal Lecture # 07 Spring-2016.
EMI INFSO-RI Catalogue synchronization & ACL propagation Fabrizio Furano (CERN IT-GT)
Implementation of GLUE 2.0 support in the EMI Data Area Elisabetta Ronchieri on behalf of JRA1’s GLUE 2.0 Working Group INFN-CNAF 13 April 2011, EGI User.
EMI INFSO-RI EMI 1 (Kebnekaise) Updates C. Aiftimiei (INFN) EMI Release Manager.
UNICORE and Argus integration Krzysztof Benedyczak ICM / UNICORE Security PT.
EMI is partially funded by the European Commission under Grant Agreement RI EMI Status And Plans Laurence Field, CERN Towards an Integrated Information.
Federating Data in the ALICE Experiment
Argus EMI Authorization Integration
Outline Introduction and motivation, The architecture of Tycho,
DAaM summary Ian Bird MB 29th June 2010.
EGEE Middleware Activities Overview
Peer-to-peer networking
A Messaging Infrastructure for WLCG
John Gordon (STFC) APEL PT Leader
Replication Middleware for Cloud Based Storage Service
Messaging Services and Client Software
Presentation transcript:

EMI INFSO-RI Catalogue synchronization & ACL propagation Fabrizio Furano (CERN IT-GT)

EMI INFSO-RI The problem The Working Group Goal and milestones Message brokers Communication protocols and guidelines Conclusions 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Outline

EMI INFSO-RI Various catalogues keep information that is related – E.g. LFC keeps info about the content of remote SEs, each one with its catalogue – A change in the permissions of a file in LFC is not automatically reflected by the peripheric catalogue – If a SE looses a file, the LFC des not know Keeping them in sync is a very hard problem 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The problem

EMI INFSO-RI Actually, the updates to the various catalogues are performed as external actions – By the jobs, as long as they do not die/crash in the wrong moment – As scheduled rounds of massive synchronizations 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The problem

EMI INFSO-RI Current consistency model is not resilient to failures – Storage failures lead to dangling entries to be cleaned up manually. Catalogue failures lead to orphaned files. Namespace scanning for diffs is an expensive workaround 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The problem

EMI INFSO-RI Make the various catalogues/SE able to talk to each other – In order to exchange messages that keep them synchronized – 2 ways: LFC->SE to propagate changes in the permissions SE->LFC to propagate info about lost files – These are just starting examples, what can be done can be even much more interesting 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The proposal

EMI INFSO-RI In other words: – Disasters and misalignments will still happen Nobody can prevent a file from disappearing – Let’s design a mechanism that ruggedizes the catalogues by means of notifications – Let’s design a mechanism to propagate the changes in the LFC to all the interested SEs 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The proposal

EMI INFSO-RI The kickoff mail was sent a few days ago in emi-jra1-data Working group: – Jean-Philippe B. – Paul M. – Riccardo Z. – Michele D. – Fabrizio F. (Me) 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation The working group

EMI INFSO-RI Proposed demonstrators to use reliable message (i.e. industry standard MQ) as backbone of the reliability – All interested catalogues can “subscribe” for permissions that changed in the LFC – Lost files can be broadcast on the “lost” topic to interested catalogues Somehow possible also for corrupted “bad” files (e.g. not readable) 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Milestone

EMI INFSO-RI Looking for good ways to reliably communicate and cooperate Communication 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation SE1 SEn SE2 SE or exp. catalogue subscribes to the relevant topics (e.g. “Changes”) SE sends to the appropriate topics (e.g. “Lost”) LFC Subscribes to the relevant topics (e.g. “Lost”) SE Sends to the appropriate topics (e.g. “Changes”) LFC ATLAS catalogue ATLAS catalogue Broker(s)

EMI INFSO-RI Industry standard MQ brokers, like ActiveMQ will constitute the backbone Need to: – Agree on the needed messages (content+semantics) – Decide the implementation guidelines 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Message brokers

EMI INFSO-RI Several APIs are available, usable from nearly every language Each API talks to the broker using a protocol. Mostly, in our case: – OpenWire (binary) – STOMP (XML based) The broker speaks them all and assures the interoperability The idea is not to enforce one protocol, just give usage guidelines 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation APIs and protocols

EMI INFSO-RI Although compatible, each API/Protocol has its own idiosyncrasies. E.g.: The recent release of the C++ library is simply broken. I had to do all the tests with (their) trunk, with very good results. The reference STOMP implementation that I tested does not support recovery. Any glitch silently makes a zombie client until manual restart. Probably many others. – Need to sort out the needed features 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation APIs and protocols

EMI INFSO-RI Many aspects have still to be sorted out (E.g. security requirements), but at least we started well. Making catalogues and SEs interact seems a good way to solve the consistency problem The messaging broker tech is not new, at least we know that it works The performance seems good, eventually it should also be sufficiently scalable Next goal: demo for December Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation Conclusions

EMI INFSO-RI Thank you 13 Oct 2010 F.Furano - Catalogue Synchronization & ACL propagation EMI is partially funded by the European Commission under Grant Agreement INFSO-RI