1 End-host Route Selection in the CHEETAH Networking Solution Zhanxiang Huang 05/01/2006 Advisor: Malathi Veeraraghavan Master’s Project Presentation Acknowledgement: This work was carried out under the sponsorship of NSF ITR , NSF ANI , NSF ANI , and DOE DE-FG02-04ER25640 grants.

2 Outline CHEETAH project overview End-host route selection problem Model-based solution Measurement-based solution Conclusion and future work

3 Circuit-switched High-speed End-to-End Transport ArcHitecture (CHEETAH) Connectionless Best-effort Internet Goal: high-speed rate-guaranteed end-to-end circuits with call-by-call-based bandwidth sharing long term leased line (under-utilized & expensive) Telephony Network 64kbps circuits end-to-end connection Congestion Delay Jitter Loss

4 CHEETAH Applications Applications: –video telephony –high speed file transfer –remote visualization especially in eScience community, e.g. Terascale Supernova Initiative (TSI) projectTerascale Supernova Initiative (TSI) project Internet

5 Current CHEETAH Network Control card OC192 card GbE/10GbE card Cray X-1 ORNL SN16000 OC192 card Control card GbE card SN16000 OC192 card Control card Atlanta SN16000 OC192 card OC-192 GbE/10GbE card … high-speed network dynamic signaling scheme end-host software signaling engine NCSU UVA CUNY signaling engine NC GTech

6 CHEETAH End-host Software Architecture –OCS: check Optical Connection Service availability. –Routing Decision: choose between circuit and Internet path for each file transfer. –RSVP-TE Module: dynamic provision of circuits. –C-TCP: transport layer protocol optimized for circuits. Internet End-host Application TCP/IP NICII CHEETAH Network NICI RSVP-TE Module Routing Decision OCS Client CHEETAH software C-TCP NICII NICI RSVP-TE Module Routing Decision OCS Client CHEETAH software Application TCP/IP C-TCP

7 Circuit or Internet Path? Circuit setup requests may be denied. It depends on the data transfer delays on the two paths. Internet (best-effort path) CHEETAH Network (circuit) End-host Circuit transfer delay is about 5.1 seconds. Internet transfer delay is about 100ms. An extreme example: Transfer a 1K-byte file using TCP. round trip time=24ms Bottleneck link rate=100Mbps round trip time=8ms circuit rate=1Gbps setup delay=5 seconds

8 What Determines Data Transfer Delays? Over paths: –Circuit: Circuit rate Round trip time Setup delay –Internet: Round trip time Bottleneck link rate Packet loss rate At end-hosts: –Transport layer protocol and parameter settings –OS Process scheduling –Hard disk throughput

9 How to Estimate Data Transfer Delays? Model-based solution –Construct mathematical models for computing file transfer delays over the circuit and Internet paths. Measurement-based solution –Estimate file transfer delays based on delay measurements of past file transfers.

10 Model-based Solution Modeling TCP delay over Internet path –TCP Reno delay model [UMass98] Modeling delay over CHEETAH circuit –Let P b be the call blocking probability –Average delay over circuit is

11 Inputs to Delay Models Inputs to TCP Reno delay model: –File size –Bottleneck link rate –Round trip time –Packet loss rate –Initial congestion window size –Sender and receiver buffer sizes Inputs to circuit delay model: –File size –Circuit rate –Round trip time over the circuit path –Round trip time over the signaling path –Call processing delay at each switch –Signaling engine call load –Number of switches on the path –Call blocking probability

12 Limitations of the Model-based Solution Packet loss rate is difficult to measure. (Tools that I tested include Sting, iperf, ping, badabing and etc.) Same are call blocking probability and signaling engine call load. Many TCP variants are emerging but there is no delay model for them yet. –e.g. BIC-TCP has been included in linux kernel 2.6 but has not been modeled yet.

13 Internet Measurement-based Solution Assumptions –Fixed circuit rates, e.g. 1Gbps, 100Mbps… –The number of destinations with which an end-host typically communicates, is not large. –Internet traffic has repeating patterns over time, which means that during a specific time period, round trip time, packet loss rate and call blocking probability are likely the same. delay file size circuit Internet 0 crossover circuit Idea: Discretize time and file size, at each time slot, for each destination and each circuit rate, measure the delays of file transfers over both paths to find the crossover file size.

14 Active and Passive Measurements Active measurements –Traffic is injected into the network explicitly for the purpose of obtaining measurements. Passive measurements –Data is collected under normal network usage.

15 A Best-case Active-measurement Experiment Best-case means packet loss rate and call blocking probability are equal to zero. TCP buffers are set to Bandwidth Delay Product values. Drawback: significant measurement traffic overhead

16 mid Active Measurements Delays on Internet path and circuit are random variables, D I and D C. 1.Find an interval (min, max) that contains the crossover file size; 2.Measure delays on both paths for file size mid=(min+max)/2; 3.If |E(D I )-E(D C )|<e, crossover=mid; 4.If E(D I )>E(D C ), max=mid; 5.If E(D I )<E(D C ), min=mid; 6.Go to 2; delay file size circuit Internet 0 crossover min max Drawback: measurement traffic overhead Let M be the initial max file size and N be the initial min file size. Traffic size = O(M*log(M-N)).

17 Passive Measurements 1.Initiate (min, max) with (0, +inf). 2.If file size < min, choose Internet; 3.If file size > max, choose circuit; 4.If min <= file size <= max, choose each path with probability ½. Record the data transfer delays. 5.Once there are sufficient records to compute Pr(D I - D C >0) for a file size in (min, max), adjust min or max based on Pr(D I -D C >0). p file size max min 0 1 1/2 crossover (Note that min and max are file sizes in application queries and assume D I and D C follow normal distributions.)

18 Hybrid Measurements Fast startup –Find the bottleneck link rate of the Internet path and the circuit setup delay through either passive or active measurement. –Solve the equation for “file_size”. –Init (min, max) with (file_size/2, file_size*2). Use active measurements when initiated by administrator users.

19 Bookkeeping Data Structure Time SlotDestinationCircuit Rate Crossover File Size Transfer Delay Records File SizeD I (sec) D C (sec) 02:00 – 03:00 Sunday Gbps50MByte – 70MByte 50MByte MByte MByte ……… …

20 Interaction Between CHEETAH Software Modules and Applications

21 Evaluation Experiment setup –The Routing Decision server and an application run on a Linux-2.6 box with 2 Xeon 2.8GHz CPUs and 1GB memory. –The application queries with parameters,. The database has an entry corresponding to this IP and time slot. –Internet path: bottleneck link rate=100Mbps; round trip time =24ms. Circuit: round trip time=8ms. Delay –An application submits 100 queries. –Mean query delay = sec < round trip time << 5 sec (the average setup delay). –Query delay standard deviation = e-004 sec < 0.3ms

22 Conclusion and Future Work Conclusion –Measurement-based solution is better than the model- based solution.  Adaptive to new TCP variants  Adaptive to the traffic pattern changes  Adaptive to hardware or software configuration changes  Low overhead Future work –Scalability issues For a computer that communicates with a large number of end-hosts (e.g. a web server), we can separate the RD module from the computer and run a separate RD server for it. For computers in the same LAN and with the same hardware and software configurations, we create an RD server for the whole LAN.

23 Reference [CHEETAH] M. Veeraraghavan, X. Zheng, H. Lee, M. Gardner, W. Feng, CHEETAH: Circuit-switched High-speed End-to-End Transport ArcHitecture, Proc. of Opticomm 2003, Oct , Dallas, TX, Won Best Student Paper Award.CHEETAH: Circuit-switched High-speed End-to-End Transport ArcHitecture, [C-TCP] A. P. Mudambi, X. Zheng, and M. Veeraraghavan, A Transport Protocol for Dedicated End-to-End Circuits, accepted by ICC 2006.A Transport Protocol for Dedicated End-to-End Circuits [UMass98] J. Padhye, V. Firoiu, D. Towsley and J. Kurose. Modeling TCP throughput: A simple model and its empirical validation. In SIGCOMM ’98, September 1998.

24 Backup Slides

25 How to compute Pr(D I -D C >0)? Assume the delays observed on the Internet path and the circuit are normally distributed random variables, D I and D C. Each file size has these two random variables. 0 E(D I -D C ) P(D I -D C ) D I -D C

26 CHEETAH network Centuar FastIron FESX448 1G Compute Orbitty Compute Nodes 1G OC192 GbE MCNC Catalyst 7600 H H H H H 1G Compute Compute Compute Compute G Wukong cheetah-nc 3x1G VLAN OC GbE GbE G Zelda H H H 1G Zelda Zelda G Zelda H H Zelda x1G MPLS tunnels 1G Cheetah-atl OC-192 lamda 10GbEGbE Cheetah-ornl OC192 X1(E)UCNS 1GFC 1G Juniper T320 Juniper T320 1G Force10 E300 switch ORNL Atlanta NC Direct fibers VLANs MPLS tunnels Wuneng H H 1G UVa Catalyst 4948 WASH HOPI Force10 WASH Abilene T640 NCSU M20 2x1G MPLS tunnels CUNY Foundry NYC HOPI Force10 1G UVa host H CUNY host H 1G UVa CUNY By Xuan Zheng,

27 Delay model

28 Circuit delay model (1)

29 Circuit delay model (2)

30 TCP-Reno delay model (1)

31 TCP-Reno delay model (2)

32 TCP-Reno delay model (3)

33

34 Measurement example room in

35 Experiment setup mvstu6 CPU 2 CPUs, each is Intel(R) Xeon(TM) CPU 2.80GHz with 1024KB cache Memory 1GB Hard disk 1 MegaRAID Model: LD 0 RAID0 69G OS _FC3smp File system EXT3 NIC Intel PRO/1000 Single Port Adapters working at rate 100Mbps, Full Duplex

36 Acronym CHEETAH – Circuit-switched High-speed End- to-End Transport ArcHitecture PLR – Packet Loss Rate SD – Setup/Teardown Delay RTT – Round Trip Time AB – Available Bandwidth GMPLS – Generalized Multiple Protocol Label Switching SONET – Synchronous Optical NETwork SDH – Synchronous Digital Hierarchy