1 SRM-Lite: overcoming the firewall barrier for large scale file replication Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory April, 2007

2 First, some background: What are SRMs Storage Resource Managers (SRMs) are middleware components whose function is to provide:Storage Resource Managers (SRMs) are middleware components whose function is to provide: dynamic space allocation AND file management in spaces for storage components on the local or wide-area network Based on a common standard SRM (BeStMan) client/user applications Unix-based Disk Pools Examples of some storage systems currently supported by SRMs dCache CASTOR CCLRC RAL GPFS SRM (DPM) SRM/ dCache SRM/ CASTOR SRM (StoRM) Unix-based Disk Pools

3 SRM Functional Concepts Manage Spaces dynamically Reservation, lifetime Negotiation Manage files in spaces Request to put files in spaces Request to get files from spaces Lifetime, pining of files, release of files No logical name space management (done by replica location services) Access remote sites for files Bring files from other sites and SRMs as requested Use existing transport services (GridFTP, https, …) Transfer protocol negotiation Manage multi-file requests Manage request queues Manage caches Manage garbage collection Directory Management Uxix semantics: srmLs, srmMkdir, srmMv, srmRm, srmRmdir

4 Tomcat servlet engine Tomcat servlet engine MCS Metadata Cataloguing Services MCS Metadata Cataloguing Services RLS Replica Location Services RLS Replica Location Services SOAP RMI MyProxy server MyProxy server MCS client RLS client MyProxy client GRAM gatekeeper GRAM gatekeeper CAS Community Authorization Services CAS Community Authorization Services CAS client disk MSS Mass Storage System HPSS High Performance Storage System disk HPSS High Performance Storage System disk DRM Storage Resource Management DRM Storage Resource Management HRM Storage Resource Management HRM Storage Resource Management HRM Storage Resource Management HRM Storage Resource Management HRM Storage Resource Management HRM Storage Resource Management gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server openDAPg server openDAPg server gridFTP Striped server gridFTP Striped server LBNL LLNL ISI NCAR ORNL ANL DRM Storage Resource Management DRM Storage Resource Management Example Use of SRMs in Earth Science Grid (in production for 3 years) 3100 users120 TBs managed LBNL’s SRMs inter-communicate between several sites and main portal site at NCAR to provide storage management and multi-file movement

5 DataMover: SRMs used to provide large scale robust data streaming between sites Problem: move thousands of files robustlyProblem: move thousands of files robustly Takes many hours Need error recovery Mass storage systems failures Network failures Solution: Use Storage Resource Managers (SRMs)Solution: Use Storage Resource Managers (SRMs) File streaming paradigm By reserving and releasing storage space automatically Problem: too slowProblem: too slow Solution:Solution: in GridFTP Use parallel streams Use large FTP windows Pre-stage files from MSS Use concurrent transfers NERSC Anywhere BNL Disk Cache Disk Cache SRM-COPY (thousands of files) SRM-GET (one file at a time) DataMover SRM (performs writes) SRM (performs reads) GridFTP GET (pull mode) stage files archive files Network transfer Get list of files Example setup for STAR high-energy-physics experiment

6 SRM-Lite Goal 1: automate file movement behind a firewall a client program to automate movement of multiple files to/from client’s directory to a remote site given a OTP firewall at one site Support entire directory transfers Recover from mid-transfer interruption and machine failure Goal 2: pull files into user’s workstation Use SRM-Lite by users to download files into their workstations Using various transfer protocols (GridFTP, bbcp, https, …) Have a GUI that shows transfer progress Or have a command line Support entire directory transfers Support suspend/resume operations (e.g. on laptops)

7 SRM-Lite: a client program to automate movement of multiple files to/from client’s directory to a remote SSH server given a OTP firewall at one site Disk Cache HPSS SSH Server NERSC SSH Request GridFTP/FTP/ SCP transfers Process StepsProcess Steps Login to ORNL using OTP At ORNL invoke SRM-Lite User composes XML input file, srmlite.xml for selected files/directories to copy from/to another site Or, user gives command line option for a selected file/directory SRM-Lite uses srmlite.xml or command line input to automatically Push/Pull files to/from NERSC Use multiple threads for concurrent transfers Disk Cache ORNL SRM- Lite OTP Login Use Case A: OTP firewall at local site (ORNL), SSH server at remote site (NERSC) srmlite.xml Local Commands

8 Scenario: one end has SRM, The other end has a firewall, use SRM Disk Cache HPSS SRM NERSC GridFTP/FTP/ SCP transfers Process StepsProcess Steps Login to ORNL using OTP At ORNL invoke SRM-Lite User composes XML input file, srmlite.txt for selected files/directories to copy over to/from another SRM controlled storage system Or, user gives command line option for a selected file/directory SRM-Lite uses srmlite.xml or command line input to automatically Push/Pull files to/from SRM at NERSC Use multiple threads for concurrent transfers Disk Cache ORNL SRM- Lite OTP Login srmlite.xml SRM Request Use Case B: OTP firewall at local site (ORNL), SRM server at remote site (NERSC)

9 Scenario: one end has SRM, The other end has a firewall, use either SSH/SRM, Disk Cache HPSS SRM SSH Server NERSC SSH Request GridFTP/FTP/ SCP transfers Process StepsProcess Steps Login to ORNL using OTP At ORNL invoke SRM-Lite User composes XML input file, srmlite.xml for selected files/directories to copy over to another site Or, user gives command line option for a selected file/directory SRM-Lite uses srmlite.xml or command line input to automatically Push/Pull files to/from NERSC using either SSH or SRM Use multiple threads for concurrent transfers Disk Cache ORNL SRM- Lite OTP Login srmlite.txt SRM Request Use Case C: OTP firewall at local site (ORNL), SRM/SSH server at remote site (NERSC)

10 Scenario: both ends have SRMs, both ends have a firewall, use SRM-lite on both ends, Use SSH to invoke SRM-Lite at other end Process StepsProcess Steps Login to ORNL using OTP Create a OTP SSH tunnel to NERSC User composes XML input file, srmlite.txt for selected files/directories to copy over to another site Or, user gives command line option for a selected file/directory SRM-Lite uses srmlite.txt or command line input, through SSH tunnel to automatically Communicate with SRM-lite at other end Push/Pull files to/from NERSC using SRM Use multiple threads for concurrent transfers Can use SCP onlyCan use SCP only NERSC SCP transfers ORNL OTP Login SSH-Tunneling OTP SSH Tunnel Disk Cache SRM- Lite srmlite.txt HPSS SRM Disk Cache HPSS SRM SRM- Lite SRM Request Use Case D: OTP firewall at both local site (ORNL), and remote site (NERSC), Use SRM-lite at both ends

11 SRM-Lite: Status SRM-Lite is developed Available from: Tested with GridFTP, SCP, HTTPS, HTTP Tested with large number of files Tested behind a firewall Access from local SRMs that acess HPSS – not tested yet Access between two filewalled system – not developed yet

12 SRM-Lite: GUI GUI was developed when used at user’s site (Linux, PC, MAC), called DataMover-lite (DML) Available from: Example GUI screen Shows info on: completed, active, and pending transfers Also, file sizes, transfer times, transfer speed

13 Extra Slides

14 Storage Resource Managers SRMs are middleware components whose function is to provide:SRMs are middleware components whose function is to provide: dynamic space allocation AND file management in spaces for storage components on the local or wide-area network Based on a common standard SRM (BeStMan) JASMine client/user applications Unix-based disks Examples of storage systems currently supported by SRMs dCache CASTOR CCLRC RAL GPFS SRM (DPM) SRM (Jlab-SRM) Unix-based disks CASTOR SRM (StoRM) SRM/ dCache SRM/ CASTOR SRM/ CASTOR SRM (StoRM) MSS SRM/ L-Store SRM (BeStMan) SRM (BeStMan)

15 DataMover-Lite use in ESG: a client program used to automate movement of multiple files to client’s directory Disk Cache MSS SRM ESG Portal Disk Cache User’s browser DataPortal NCAR User’s machine DataMover Lite release request GridFTP/FTP/ HTTP/HTTPS transfers Process StepsProcess Steps User downloads DataMoverLite User goes to portal, select files Portal gets ALL files into SRM disk Portal generates XML input file, datamover.txt, for user selected files DML uses datamover.txt to automatically get files, and release files after move completes successfully datamover.txt

16 Another example of DML GUI