1. lundi 25 février 2019
FTS configuration
David Bouvet (CC-IN2P3) Lionel Schwarz (CC-IN2P3) Jonathan Schaeffer (CC-IN2P3)

2. The File Transfer Service (FTS) is a data movement fabric service
FTS overview
The File Transfer Service (FTS) is a data movement fabric service
It is a multi-VO service, used to balance usage of site resources according to VO and site policies
For management ease, the service supports splitting jobs into multiple “channels”
Once a job is submitted it is assigned to a suitable channel for serving

3. Channel concept A channel may be: Channels are uni-directional
A point to point network link
Various “catch-all” channels
(e.g. anywhere to me, or me to anywhere)
Channels are uni-directional
e.g. : 2 channels : IN2P3-CPPM and CPPM-IN2P3
All file transfer jobs on the same channel are served as part of the same queue
Intra-VO priorities for the queue (Atlas gets 75%, CMS gets the rest)
Internal-VO priorities within a VO
Each channel has its own set of transfer parameters
Number of concurrent files running, number streams, TCP buffer, etc

4. FTS architecture Experiments interact via web-service
VO agents do VO-specific operations (1 per VO)
Channel agents to channel specific operation (e.g. the transfers)
Monitoring and statistics can be collected via the DB

5. How it works ? A user can submit any transfer request to an FTS server
i.e. with any source / destination SURL pairs
e.g. “srm://ccsrm.in2p3.fr/pnfs/in2p3.fr” to “srm://lapp-se01.in2p3.fr/dpm/in2p3.fr”
The server will resolve the sites from the SURLs
e.g. “IN2P3-CC” to “IN2P3-LAPP”
It then looks for a point-to-point channel defined on that FTS server that it able to serve that site pair
If it cannot find one, it fails the job immediately
Otherwise it assigns the job to the channel’s queue and the job is served according to the channel parameters
The more specific channels are always matched first if they are defined

6. FTS deployment convention
A tier-1 should manage any transfer for which it is the destination (except for T0 export)
Focus is to avoid having to run an FTS server at the T2
Associated T2 to T1 is run from the T1
Suggested convention is that the T1 site FTS server is responsible for all transfers where one of its associated T2s is the destination.
T1 defines a single “any to my tier-2s channel”
And can define specific channels if it wishes to manage any of them separately

7. Channels configuration
Two types of channels supported by FTS :
“URLCOPY” – normal FTS 3rd party copy (SRM negociation with 3rd party gridFTP)
“SRMCOPY” – SRM copy (delegate copy to SRM)
DPM does not supporte “SRM copy” mode  channels are configured following destination SE type
“URLCOPY” for DPM SE or unknown SE (e.g. for catch-all channels)
“SRMCOPY” for dCache SE

8. Channels configuration
FTA_STAR_LAPP="URLCOPY"
FTA_STAR_LPC="URLCOPY"
FTA_STAR_CPPM="URLCOPY"
FTA_STAR_TOKYO="URLCOPY"
FTA_STAR_BEIJING="SRMCOPY"
FTA_IN2P3_BELGIUMULB="SRMCOPY"
FTA_BEIJING_IN2P3="SRMCOPY"
FTA_BELGIUMUCL_BELGIUMULB="SRMCOPY"
FTA_BELGIUMUCL_IN2P3="SRMCOPY"
FTA_BELGIUMULB_BELGIUMUCL="SRMCOPY"
FTA_BELGIUMULB_IN2P3="SRMCOPY"
FTA_BNL_IN2P3="SRMCOPY"
FTA_CNAF_IN2P3="SRMCOPY"
FTA_CPPM_IN2P3="SRMCOPY"
FTA_GRIF_IN2P3="SRMCOPY"
FTA_GRIDKA_IN2P3="SRMCOPY"
FTA_IN2P3_BEIJING="SRMCOPY"
FTA_IN2P3_BELGIUMUCL="SRMCOPY"
FTA_IN2P3_ITEP="URLCOPY"
FTA_IN2P3_TOKYO="URLCOPY"
FTA_IN2P3_CPPM="URLCOPY"
FTA_IN2P3_GRIF="URLCOPY"
FTA_IN2P3_IN2P3="SRMCOPY"
FTA_IN2P3_LAPP="URLCOPY"
FTA_IN2P3_LPC="URLCOPY"
FTA_IN2P3_SUBATECH="URLCOPY"
FTA_ITEP_IN2P3="SRMCOPY"
FTA_LAPP_IN2P3="SRMCOPY"
FTA_LPC_IN2P3="SRMCOPY"
FTA_PIC_IN2P3="SRMCOPY"
FTA_RAL_IN2P3="SRMCOPY"
FTA_SARA_IN2P3="SRMCOPY"
FTA_STAR_BELGIUMUCL="SRMCOPY"
FTA_STAR_BELGIUMULB="SRMCOPY"
FTA_STAR_GRIF="URLCOPY"
FTA_STAR_IN2P3="SRMCOPY"
FTA_STAR_ITEP="URLCOPY"
FTA_SUBATECH_IN2P3="SRMCOPY"
FTA_TAIWAN_IN2P3="SRMCOPY"
FTA_TOKYO_IN2P3="SRMCOPY"
FTA_TRIUMF_IN2P3="SRMCOPY"
FTA_FNAL_IN2P3="SRMCOPY"

10. spare slides

11. General “catch-all” channels
Anything to me
e.g. STAR-IN2P3
Me to anything
e.g. IN2P3-STAR
This will match any site on the “STAR”
All the jobs running on the channel are managed together
If you stop the channel, they are all paused
If you reduce the #files on the channels, they will be served at a slower rate

12. Specific site channels
e.g. IN2P3 to RAL
e.g. TOKYO to IN2P3
These are unidirectional, and allow a specific flow to be managed separately from the rest
The more specific channels are always matched first if they are defined

13. T1-T1 deployment convention
A tier-1 should manage any transfer for which it is the destination (except for T0 export)
Basic mode: define a single “star to me” channel
We can manage all inbound transfers on one single channel

14. T2/T1 suggested convention
Focus is to avoid having to run an FTS server at the T2
Associated T2 to T1 is run from the T1
Suggested convention is that the T1 site FTS server is responsible for all transfers where one of its associated T2s is the destination.
CNAF is responsible for any transfers to Bari
IN2P3 is responsible for any transfers to BEIJING
Transfers go point to point – they are not routed over the FTS hierarchy
T1 defines a single “any to my tier-2s channel”
And can define specific channels if it wishes to manage any of them separately