GridTorrent Framework: A High-performance Data Transfer and Data Sharing Framework for Scientific Computing

Outline Introduction Motivation PTCP GridTorrent Framework Test Results LAN Test Results WAN Test Results Overhead Questions

Introduction Today’s computational science is data-intensive Large Hadron Collider (LHC) experiment at CERN generates petabytes of data Accessibility, replication and creation of the data are made very easy by Internet Computational Grid

Motivation The data is geographically distributed Users are dispersed Collaboration environments are required Resources should be used in efficient and effective way CPU Storage Network

PTCP TCP has a performance problem over WANs It was developed to solve the above problem by using striping technique

GridTorrent Framework It is aimed to provide collaboration environment for dispersed users to make data transfer, management, and sharing easy via content manager to use systems resources efficiently and effectively by harnessing P2P (Bittorrent) network structure

Collaboration & Content Manager The Content Manager allows users to publish or share their files with selected access control rights The Collaboration Manager permits users to build a virtual sharing environment by managing working groups or friend list ACL enforce access control rights for a given content Task Manager handles the users’ task list

GridTorrent Client It is responsible for initiating actual data publishing data sharing with other GTF clients ensuring secure environment for the above activities

WS-Tracker WS-Tracker is a WS enabled server It assists in the communication between peers (GridTorrent clients) It delivers task lists which is generated by users to GridTorrent clients It supplies ACL of each shared file to GridTorrent clients

Experimental Results A B C PTCP and GridTorrent Framework tests cases were conducted both in LAN and WAN type of computer networks Server and client machines’ specification and location table File size is 300 MB Name Specification Network Interface Institution Location A Intel(R) Quad-Core Xeon(TM) 4x2.33GHz CPU with 8GB of RAM on Red Hat Enterprise Linux 4.0 Broadcom NetXtreme II BCM5708 1000Base-T Indiana University Bloomington, IN B Sun Fire V880 8x1.2GHz UltraSPARC III processors with 16GB of RAM on Solaris 9. It has 6x72GB 10K rpm internal HD Gigabit Ethernet and 10/100-BaseT Ethernet C Dual Pentium III 731MHz CPU with 512MB of RAM on GNU/Linux 2.6.20-1.2316.fc5 Florida State University Tallahassee, FL

LAN Test Setup Server is located at Bloomington, IN Client is at Indianapolis, IN The number of parallel TCP streams between server and client has increased from 1 to 16 (PTCP) The number of seeders increased from 1 to 16 (GTF) Client and server configuration for PTCP GridTorrent test case configuration for LAN test. Regular Java sockets are used for data transfer.

LAN Test Result There is no significant improvement in bandwidth for both PTCP and GTF Experimental data transfer (80-100 Mbps) rate is much lower the theoretical (1000Mbps) and measured data transfer rate (857Mbps)

WAN Test-I Setup Server is located at Bloomington, IN Client is at Tallahassee, FL The number of parallel TCP streams between server and client has increased from 1 to 16 (PTCP) The number of seeders increased from 1 to 16 (GTF) Client and server layout for PTCP test case GridTorrent test case configuration for wide area network test. Regular Java sockets are used for data transfer.

WAN Test-I Result Bandwidth usage is vastly improved in both GTF and PTCP PTCP’s bandwidth utilization rate has risen steadily until fifteen streams its peak value is 118 Mbps GTF’s bandwidth utilization rate has risen steadily until thirteen streams GridTorrent is performing better than PTCP when parallel streams number is less than five

WAN Test-II Setup Server is located at Bloomington, IN Client is at Tallahassee, FL The number of parallel TCP streams between server and client has increased from 1 to 16 (PTCP) Besides Java socket, other data transfer protocols can be exploited in GridTorrent client The number of seeders increased from 1 to 16 (GTF) Four parallel TCP sockets were used between peer and seeders Client and server layout for PTCP test case GridTorrent test case configuration for wide area network test. GridTorrent client uses four parallel TCP sockets in each connection for every source

WAN Test-II Result Using parallel TCP with Bittorrent algorithm demonstrates much better bandwidth usage than standalone GridTorrent and PTCP The maximum attained bandwidth is around 145 Mbps which is %23 higher than PTCP’s result

Overhead Both parallel TCP and GridTorrent have overhead due to nature of multiple parallel connections PTCP’s communication channel overhead time can be compared to GridTorrent WS-Tracker client’s overhead time varying between 300 and 600 milliseconds Another overhead of GridTorrent is that control messages exchanged between peers to ensure strictly enforced to all participating peers The total size of overhead messages is between 148KB to 169 KB

Questions

Thanks to All.