1. WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing

2. Overview  The FTS3 service  WebFTS features  WebFTS cloud integrations  Dropbox  CERNBox  Ongoing development  Dropping X509  Data management operations 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 2

3. What is WebFTS?  Web based tool to transfer files between grid/cloud storages  Modular protocol support  gsiftp, http(s), xrootd and srm  Cloud extensions: dropbox, CERNBox  Development funded by 17/11/2014 3 WebFTS: File Transfer Web Interface for FTS3

4. Provide access to leading technology Based on FTS3 FTS3 is the service responsible for distributing the majority of LHC data across the WLCG infrastructure Low level data movement service, responsible for moving sets of files from one site to another while allowing participating sites to control the network resource usage Used by LHC VOs + many others VOs part of EGI ~20PB monthly transfer volume / ~2.2M files per day (WLCG) http://dashb-fts-transfers.cern.ch/ui/ http://dashb-fts-transfers.cern.ch/ui/ 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 4

5. WebFTS architecture WebFTS: File Transfer Web Interface for FTS3 BROWSER REST API GFAL2 FTS3 DAVIX GSIFTP DROP BOX … … WEBFTS.js 17/11/2014 5

6. Security  Simpler access while keeping the same level of security  X509 + Oauth for Dropbox  Transparent delegation of credentials  Avoid storing or transferring any sensitive data  Open access to all source code  All sensitive information is used within the browser and forgotten 17/11/2014 6 WebFTS: File Transfer Web Interface for FTS3

7. Delegation  Delegation is needed to let WebFTS access the grid on users behalf  Users make private key available to browser  Not available via browser API  VOMS extensions acquired by the service on users behalf  Why it’s important  Gives the users a service which can access the grid for them, from a browser, with full VOMS credentials WebFTS: File Transfer Web Interface for FTS3 17/11/2014 7

8. Additional Features  Check-summing and file overwriting  Possibility to resubmit transfer jobs or only-failed files transfers.  Storage Endpoints Auto-completion  For endpoints published on the BDII (EGI and WLCG Information System)  Support for LFC Registration  File catalog developed at CERN and used by EGI and WLCG 17/11/2014 8 WebFTS: File Transfer Web Interface for FTS3

9. Success Stories  WebFTS has been successfully tested to transfer from/to:  EUDAT B2Stage ( iRODS DSI)  Any gsiftp/webdav/xrootd aware grid storage ( DPM, dCache, Castor, EOS, Storm)  HPC Titan @ Oak Ridge National Lab (ongoing)  https://www.olcf.ornl.gov/titan/ https://www.olcf.ornl.gov/titan/  Under evaluation by LHCb 17/11/2014 9 WebFTS: File Transfer Web Interface for FTS3

10. Landing page and Guided-tour 17/11/2014 10 WebFTS: File Transfer Web Interface for FTS3

11. Credential delegation 17/11/2014 11 WebFTS: File Transfer Web Interface for FTS3

12. Transfer interface 17/11/2014 12 WebFTS: File Transfer Web Interface for FTS3

13. Job status interface 17/11/2014 13 WebFTS: File Transfer Web Interface for FTS3

14. Extension for Dropbox  Nice way import/export data from the grid world  Avoid the installation of new software and uses what the user has already installed  Zero development of clients  Multiplatform is given for free  Integration with Oauth  By delegating to FTS the right to interact with dropbox on users behalf  Achieved using web tech  Which requires the interactivity of a browser WebFTS: File Transfer Web Interface for FTS3 17/11/2014 14

15. 17/11/2014 15 WebFTS: File Transfer Web Interface for FTS3 Extension for Dropbox

16. Dropbox plugin  Server side the development of a plugin for the metadata management and I/O operations was needed:  FTS REST integrates the plugin to perform metadata management operations  FTS3 server uses the plugin to perform the transfers:  GridFTP dropbox  Http(s) dropbox WebFTS: File Transfer Web Interface for FTS3 17/11/2014 16

17.  While Dropbox has been integrated via the implementation of a plugin for CERNBox we waited for the new version with EOS as backend ( CERNBox 2.0)  We use EOS access via standard grid protocols ( e.g. xrootd)  We map user credentials to correct EOS namespace  The rest comes for free CERNBox integration 17/11/2014 17 WebFTS: File Transfer Web Interface for FTS3

18. WebFTS With CERNBox 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 18

19. 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 19 WebFTS With CERNBox

20. 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 20 WebFTS With CERNBox

21. 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 21 WebFTS With CERNBox

22. 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 22 WebFTS With CERNBox

23. 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 23 How can we get rid of the delegation step? 1. An Identity Federation: eduGAIN  To allow identity providers to authenticate users at their own institute (SSO) 2. A token translation service : STS  To ask the CA for a certificate for the users 3. An “IOTA” Certification Authority  To grant the short lived certificate 4. VOMS  To accept the new cert as a VO member Ongoing developments: Access without X509

24. EDUGAIN WebFTS: File Transfer Web Interface for FTS3  Built on existing federations and infrastructures  CERN participates in eduGAIN via SWITCHaai  Many NRENs participate in eduGAIN too 24 17/11/2014 24

25. Security Token Service (STS) WebFTS: File Transfer Web Interface for FTS3 An EMI service SAML in, X509/VOMS out 17/11/2014 25

26. “IOTA” CA WebFTS: File Transfer Web Interface for FTS3 17/11/2014 26

27. VOMS admin WebFTS: File Transfer Web Interface for FTS3 17/11/2014 27

28. Architecture WebFTS CERN SSO IdP Credentials Attributes Web Redirect WAYF SAML VOMS IdP Grid Storage Element Grid Storage Element X.509 VOMS STS IOTA CA IOTA CA SAML X.509 VOMS Slide adapted from Romain Wartel, GDB Sept 2014 28

29. Pros/Cons  X509-free access to the grid infrastructure  With VOMS support  Without modifying all the services  Federated single sign on  One password to remember  Numerous services potentially accessible  But we need Site acceptance WebFTS: File Transfer Web Interface for FTS3 17/11/2014 29

30.  Not only Transfers..  FTS REST API have been extended to support data management operations  Delete  Create/Remove folders  Rename  Under integration in WebFTS 17/11/2014 30 WebFTS: File Transfer Web Interface for FTS3 Ongoing developments: Data Management

31.  Online service accessible:  https://webfts.cern.ch  try now! https://webfts.cern.ch  User certificate in your browser  User guide, F.A.Q:  Online guided-tour  http://fts3-service.web.cern.ch/documentation/webfts http://fts3-service.web.cern.ch/documentation/webfts  Official support & code  fts-support@cern.ch fts-support@cern.ch  https://github.com/cern-it-sdc-id/webfts https://github.com/cern-it-sdc-id/webfts Links 17/11/2014 31 WebFTS: File Transfer Web Interface for FTS3

32. Questions? fts-support@cern.ch 17/11/2014 32 WebFTS: File Transfer Web Interface for FTS3