6 th dCache WS | Daniel Becker| 18 April 2012 | 1 Daniel Becker 6 th dCache workshop, Zeuthen, April 18, 2012 The HTTP Federation

6 th dCache WS | Daniel Becker| 18 April 2012 | 2 Content WLCG federated storage The catalogue synchronization solution Redirection using http federation Redirection on error

6 th dCache WS | Daniel Becker| 18 April 2012 | 3 How is WLCG federating storage Catalogues Storage Elements Client store register NON-ATOMIC Problem: Often results in dangling references (and “dark” data)

6 th dCache WS | Daniel Becker| 18 April 2012 | 4 Suggested solution: CAT-SYNC EMI INFSO-RI SE1 SEn SE2 SE or exp. catalogue subscribes to the relevant topics (e.g. “Changes”) SE sends to the appropriate topics (e.g. “Lost”) Subscribes to the relevant topics (e.g. “Lost”) SE Sends to the appropriate topics (e.g. “Changes”) Central Catalogue Other catalogue/SE e.g. ATLAS Broker(s) Stolen from Fabrizio Propagates “Lost File” events to catalogues

6 th dCache WS | Daniel Becker| 18 April 2012 | 5 Good for now Is a sufficient solution for now But there might be better ones

6 th dCache WS | Daniel Becker| 18 April 2012 | 6 Our Goal Web DAV Provide transparent access to data through catalogues, using standard protocols : http(s), WebDAV Redirection from catalogues to the final data source doesn’t require intermediate steps by the user but is part of the protocol.

6 th dCache WS | Daniel Becker| 18 April 2012 | 7 How does it work ‘dir’ request Or GET LFN File list or SURL User Clicks Request SURL Return TURL Quick Hack Mapping Service Redirect Head Node GET TURL Redirect GET TURL Data Pool DATA FILE FOUND Redirect SURL get request redirect Instead of File Not Found get request DATA FILE NOT FOUND NEW

6 th dCache WS | Daniel Becker| 18 April 2012 | 8 NEW: Redirection on error Global Catalogue Provides clients with encoded alternatives. Storage Elements interpret the provided alternatives. And create a redirection

6 th dCache WS | Daniel Becker| 18 April 2012 | 9 Redirection on error Transparent for the client Provides feedback to redirectors Utilizes HTTP standards only

6 th dCache WS | Daniel Becker| 18 April 2012 | 10 Redirection on error /sfn/replica/1? rid=1& r1=3, /sfn/replica/3& r2=5, /sfn/replica/5& r3=-1, /lfn/of/file

6 th dCache WS | Daniel Becker| 18 April 2012 | 11 Redirection on error /sfn/replica/3? rid=3& notfound=1& r1=5, /sfn/replica/5& r2=-1, /lfn/of/file

6 th dCache WS | Daniel Becker| 18 April 2012 | 12 Redirection on error /sfn/replica/5? rid=5& forbidden=3& notfound=1& r1=-1, /lfn/of/file

6 th dCache WS | Daniel Becker| 18 April 2012 | 13 Redirection on error /lfn/of/file? forbidden=3& notfound=1,5

6 th dCache WS | Daniel Becker| 18 April 2012 | 14 Next Step GET LFN Redirect SURL Request SURL Return TURL Quick Hack Redirect GET TURL Redirect GET TURL Mapping service Head Node Data Pool DATA Best Match Engine Portal Candidate Collection Engine Query SE’s for file e.g. http GET (header)

6 th dCache WS | Daniel Becker| 18 April 2012 | 15 Hepix (Thursday) Best Match Engine Portal Candidate Collection Engine Telecom GEO IP

6 th dCache WS | Daniel Becker| 18 April 2012 | 16 Thanks