BDTS and Its Evaluation on IGTMD link C. Chen, S. Soudan, M. Pasin, B. Chen, D. Divakaran, P. Primet CC-IN2P3, LIP ENS-Lyon

Outline Bulk Data Transfer Service A framework for Service Differentiation in Grid

Motivations Data transmission in Grid Large volume, Long term Between countable sites Between storage systems Between computation nodes and local storage system Over private shared network End-to-End quality of service. Strict deadline High reliability Best-Effort with TCP “Fair sharing” VS. Throughput intensive ASAP VS. Strict deadline Burst traffic

BDTS Bulk Data Transfer Service A centralized bandwidth allocation system Provides deadline concerned data movement End-to-End traffic control Smooth traffic Optimizes link utilization

BDTS Architecture User Interface Submit Transfer Job (t-job) Network IS Provide network information (topology, bandwidth, etc.) Job Management Optimization (BDTSh) Flow control (FLOC) Control the actual transfer according to the profile from JM

Key Concepts Network Model Static layer 2 topology and characters T-job Volume: application layer data size Time: Start time, End time Source Destination pair Path from NIS Max-rate: Limitation due to end systems Profile Time-rate: layer 2 rate.

Optimization Minimize the Congestion Factor (allocated-throughput / Link-bandwidth) Guarantee full usage of link bandwidth Reduce the average package delay experienced by the coexisting interactive traffic B. Chen and P. Primet. Schedulling deadline-constrained bulk data transfer to minimize network congestion. In CCGRID’07, Pages May 2007

Flow Control TCP + Precise Software Pacer (PSPacer) PSP A module for iproute2 Precise network bandwidth control Traffic Shaper TCP with congestion avoid mechanism disabled Optimized in restramssion and memory management Collaborating with existing TCP based application

The Estimation No Ideal Transport Protocol: Layer 2 profile VS. Layer 7 Date Transfer Protocol overhead (L7 protocol, TCP/IP header) Synchronization between the profile and implementation TCP’s re-act to the network … Linear estimation: Vs = a * Vr + b

Collaboration

The Project Job Management (Java, c++) User Interface (Java) Floc / API (C, Linux Kernel, PSP) Gridftp client (c++, globus)

Testbed

On Grid5000 GridFTP Security File transfer

Validation of System (Resource allocation)

Validation of System (Stream Isolation)

IGTMD Long latency Average 106ms Background Traffic

Conclusions Bandwidth resource management based on End-to- End traffic control Provide a deadline file transfer based on IP Open problems The effect of arriving time of t-jobs to the optimization The efficiency of Flow control The deployment of BDTS and network resource management in Grid

Outline Bulk Data Transfer Service A framework for the Service Differentiation in Grid

Grid-Managed Network Resources Why the network (bandwidth/Qos) need to be managed in a Grid running on a dedicated private Giga/Tera network? More bandwidth, more bandwidth consuming applications each member will have its own demand into the network. Grid network Applications sharing common infrastructure Middle-ware service Hosts, storage systems, networks, Network is transparent for Applications Introduce differentiation of service to application level objects?

How? Static Different quality of service is provided by different instances Applications make a choice Dynamic Service identify application Assign SLA to instance of applications Knowledge of both its users and its underlying service

BDTS and GMNR

A “ulimit” like network resource management interface Adapt networking policy to traffic according to the process it belongs to Identify Traffics from a process and its children Traffics from a user Traffics from a VO Traffics from a certain application Traffics to certain destinations Policy Output Bandwidth Diffserv QoS … Running time adjusting Inner adjusting: Grid-Job Wrapper: Deployment-User application-Uploading Result Outer adjusting Grid network Resource Management Wrapper

Thank you! Comments and Questions?