1. Fast, Scalable, and Flexible Data Sharing Made Easy
Tevfik Kosar
University at Buffalo (SUNY)
September 18th, 2017
PresQT 10th Plenary in Montreal

2. Moving a World of Data
The global IP traffic has reached an annual rate of 1.4 zettabytes, which corresponds to nearly 1 billion DVDs of data transfer per day for the entire year.
This year, more IP traffic will traverse global networks than all prior “Internet years” combined.
Terabit/sec networks, state-of-the-art HW and SW solutions… but still the fastest way to transfer data is..
FEDEX!!

3. OneDataShare
NSF-funded project (through DIBBs program), with 3 major goals:
Reduce the time to delivery of the data
Provide interoperation across heterogeneous data resources
Decrease the uncertainty in real-time decision-making processes
Implemented as a cloud-hosted service.

4. OneDataShare
OneDataShare

5. Research Challenges
Modeling the effect of a single parameter is not easy, modeling the effect of combined parameters is very challenging.

6. Parameter Optimization
Optimize:
concurrency
parallelism
pipelining
conn. caching
TCP buffer size
I/O block size
disk striping
multithreading
parallel FS
....

7. Research Challenges
Modeling the effect of a single parameter is not easy, modeling the effect of combined parameters is very challenging.
Real-time changing traffic makes any off-line modeling useless, unless it is supported by online learning and dynamic tuning.
Data sets are not homogeneous. You can have a mixture of very small files and very large files in the same dataset.
 We already do very well in addressing these challenges, and deliver up to 10X performance improvement.

8. Support for Interoperability
FTP
HTTP
SCP
GridFTP
UDT
Rsync
iRODS
SRM
BBFTP
Dropbox
FTP
HTTP
SCP
GridFTP
UDT
Rsync
iRODS
SRM
BBFTP
Dropbox

9. SW Sustainability Challenges
Maintaining the interface to many third-party software (i.e., supporting multiple data transfer protocols) is a big challenge.
Sustaining OneDataShare software development efforts beyond the terms of the NSF Award.
Sustaining the Cloud service for the cloud-hosted SW in the long term.
 We are open for suggestion on these issues!