ESA UNCLASSIFIED – For Official Use File Transfer BOF Noordwijkerhout, The Netherlands 02-Apr-14 Colin R. Haddow, ESA/ESOC Terrestrial File Transfer - Concept

File Transfer BOF| 02-Apr-14| ESA | Slide 2 ESA UNCLASSIFIED – For Official Use It’s not rocket science

File Transfer BOF| 02-Apr-14| ESA | Slide 3 ESA UNCLASSIFIED – For Official Use Purpose File exchanged between agencies may be used in, but are not limited to;  Mission design, i.e. in investigating the feasibility of a mission with respect to the support available from another agency.  Mission operations, i.e. to transfer files required for the successful operation of a mission between two or more agencies.  Post Mission activities such as archiving data related to the mission.

File Transfer BOF| 02-Apr-14| ESA | Slide 4 ESA UNCLASSIFIED – For Official Use Terrestrial File Transfer Conceptual Model Internet Agency A Firewall File Transfer Host DMZ Internal Network Agency B Firewall File Transfer Host DMZ Internal Network File Transfer

File Transfer BOF| 02-Apr-14| ESA | Slide 5 ESA UNCLASSIFIED – For Official Use Design Goals The design goals behind this concept are;  Keep the design as simple as possible whilst meeting the perceived needs  Use “off the shelf” technologies, this concept is dealing with point-to-point terrestrial file transfer, something that occurs an enormous number of times per day over the internet

File Transfer BOF| 02-Apr-14| ESA | Slide 6 ESA UNCLASSIFIED – For Official Use Protocols The protocol to be used needs to be  able to provide security for the file transfer, e.g. it should not send user ID and/or passwords in plain text (thus ruling out standard FTP).  should conform to a recognized standard  readily available on a wide range of operating systems  capable of supporting both IPv4 and IPv6 network addresses.  designed for secure data communications over an insecure network.

File Transfer BOF| 02-Apr-14| ESA | Slide 7 ESA UNCLASSIFIED – For Official Use Transport Protocols Two widely adopted transport protocols are potential candidates:  Secure Shell (SSH)  SSH is a cryptographic network protocol providing remote command-line login, remote command execution and other secure network services between two networked computers.  Transport Layer Security (TLS)  Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), are cryptographic protocols which are designed to provide communication security over the Internet. Both allow the use of certificates for login, thus making scripting of file transfer straightforward. Both are available for a wide range of operating systems

File Transfer BOF| 02-Apr-14| ESA | Slide 8 ESA UNCLASSIFIED – For Official Use Transfer Protocols Selection of transfer protocols will largely be determined by the transport protocol. Some possibilities are outlined below.  SCP and SFTP (both runs over SSH)  SCP – Secure Copy, copies files between hosts on a network and uses the same authentication and provides the same security as SSHSFTP – Secure File Transfer Program, is an interactive file transfer program, similar to FTP. It may also use many features of SSH, such as public key authentication and compression  WebDAV (runs over TLS)  Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP) that facilitates collaboration between users in editing and managing documents and files stored on World Wide Web servers. It provides a framework for users to create, change and move documents on a server; typically a web server or web share.

File Transfer BOF| 02-Apr-14| ESA | Slide 9 ESA UNCLASSIFIED – For Official Use File Types  A considerable number of the files that are likely candidates to be transferred via the TFT are likely to conform to CCSDS standards.  it is feasible that a SANA repository could be set up which would contain details of what a particular file type was, possible including the definition of the appropriate metadata for that file type.  It should however be recognized that it is also likely that a number of the files to be transferred may not conform to a CCSDS (or other recognized international standard).  The transfer of these files by means of the TFT must therefore not be precluded and it is therefore essential that the method of associating metadata with files, and the specification of the metadata, is such that this is readily expandable such that it can readily be used to transfer files that are not defined in the SANA registry.

File Transfer BOF| 02-Apr-14| ESA | Slide 10 ESA UNCLASSIFIED – For Official Use File Packaging  There are a number of different technologies readily available on a variety of platforms that could be used for wrapping the files,  ZIP  TAR  PAX  JAR  With exception of JAR none of the above provide a direct mechanism for associating metadata with the wrapped files.  Need to ensure that what is chosen can support large data volumes  Some versions of ZIP and TAR limited to approx 4Gbytes

File Transfer BOF| 02-Apr-14| ESA | Slide 11 ESA UNCLASSIFIED – For Official Use Metadata  As previously noted only JAR allows direct association of metadata with wrapped files.  Can overcome this by providing a manifest files containing the required metadata  Ways to encode required metadata include  XML – would require schema definition  XFDU –existing CCSDS standard

File Transfer BOF| 02-Apr-14| ESA | Slide 12 ESA UNCLASSIFIED – For Official Use File considerations  File and Directory naming  Permitted characters  File names  Manifest file name  Directory names  Processing  Metadata must contain any additional data required for further processing  Ensuring all files have been received  File counter in metadata ?  Further transfers  Injection into/from CFDP  Injection into/from agency specific file transfer systems

File Transfer BOF| 02-Apr-14| ESA | Slide 13 ESA UNCLASSIFIED – For Official Use Basic Operations  Required Operations  list– obtains information about a file or directory  get– retrieve a copy of the file (probably want to support wildcards, i.e. mget)  reget– reget acts like get, except that it can be used to continue a failed file transfer from the apparent point of failure.  put – store a local file on the remote machine (probably want to support wildcards, i.e. mput)  delete – delete a file on the remote machine  cd – change directory on the remote machine  mkdir – make a directory on the remote machine (Is this really needed if we define a directory structure ?)

File Transfer BOF| 02-Apr-14| ESA | Slide 14 ESA UNCLASSIFIED – For Official Use Hosts and Accounts  Hosts  Ideally the hosts for both the sender and receiver of the files would be transparently redundant, i.e. if the prime machine fails the backup machine should transparently replace it. Host names could be stored in a SANA registry.  If an approach based on TLS/WebdAV were to be adopted then, as this is an extension of http it would be possible to store the appropriate end points in a SANA registry.  Accounts  To keep things simple it is suggested that one user account per mission be foreseen. Login to this account could be by conventional user account/password. It would be however be preferable (and would certainly make automation easier) if login authentication was carried out by means of public-private key pairs. .

File Transfer BOF| 02-Apr-14| ESA | Slide 15 ESA UNCLASSIFIED – For Official Use Transfer of Files  Transfer of files  There are a number of possibilities with respect to where the transfer of files is instigated, particularly if it is assumed that the transfer of files is bi-directional, i.e. files go from one agency to a second, but files are also returned from the second agency to the first.  Simplest approach would be to assume that files are always pushed. Can we assume this ?  Is any form of notification of files being available required ?

File Transfer BOF| 02-Apr-14| ESA | Slide 16 ESA UNCLASSIFIED – For Official Use Directory Structure  Previously suggested that there is one account per mission. This being the case it is proposed that there are, within this account, only 2 directories;  in-tray – This directory would be used as the target directory for files pushed to the node. It is assumed that once a file has been found in the in-tray it will be moved to another location for further processing.  out-tray – This directory would be used either as a staging area for files that are to be pushed to the other agency or, if it is not permitted to push the files, would be where the files would be pulled from by the receiving agency. It is assumed that once a file has successfully been transferred from the out-tray it will be deleted from the out-tray or alternatively moved to an archive, in any event it will not remain in the out-tray directory

File Transfer BOF| 02-Apr-14| ESA | Slide 17 ESA UNCLASSIFIED – For Official Use Security  Not completely clear what is required but I would assume that at least the ability to optionally provide a hash value for each of the encapsulated files is  The mechanism for permitting a SHA-2 hash value for all “wrapped” files is to enclose the hash value in the meta data in the manifest file. With respect to providing the hash value for the manifest and wrapper files this can be done by appending the appropriate hash value to the full filename.  Is signing enclosed files needed ?  Is encrypting enclosed files needed ?  Is signing manifest file needed ?  Is encrypting manifest file needed ?  Is signing wrapper file needed ?  Is encrypting wrapper files needed ?

File Transfer BOF| 02-Apr-14| ESA | Slide 18 ESA UNCLASSIFIED – For Official Use Service Agreement  For the terrestrial file transfer between agencies it should be foreseen that some form of service agreement will be required. This would need to address the following points.  Points of contact  Hosts  Login Credentials  Notification Mechanism (if required)  Data Volume  Retention Time  Availability  Transfer Rate  Miscellaneous If multiple transport/transfer/wrapping mechanisms are in standard then this would indicate what is supported  Anything else ?

File Transfer BOF| 02-Apr-14| ESA | Slide 19 ESA UNCLASSIFIED – For Official Use Book Structure  With respect to defining the most aspects this should be straightforward.  Defining the required metadata for known file types is a bit trickier  Don’t want to have to issue the complete book again just because a new file type has been added or the metadata for an existing type has changed.  Put each metadata definition in a separate annex so only that annex has to be reissued if it changes ?  Put metadata definitions in SANA ?  Other alternatives ?

File Transfer BOF| 02-Apr-14| ESA | Slide 20 ESA UNCLASSIFIED – For Official Use Questions Thank you for your attention. Questions ?