1. KEK Network Qi Fazhi 2004.8

2. KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch A Netscreen Firewall Super Sinet Router 10GbE 2 x GbE IDS GbE KEK GW 2 x GbE KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch B Netscreen Firewall 2 x GbE 4 x GbE NOOD Switch to other places NOOD Switches to Buildings 4 x GbE 10GbE Tskuba Center 10GbE Other Univ. HEP Lab. GbE 4 x GbE up to: 14 Vlan, Based MAC Static & DHCP(for wireless) IDS GbE

3. KEK ’ s firewall architecture L2/L3 Switch for outside connections Central L2/L3 Switch Firewall 10GbE 10GbE/GbE 10GbE 2 x GbE Security monitor High Throughput Access Route Regular flows IDS GbE

4. KEK universities/institutes in Asia-Pacific universities in Japan SLAC, FNAL, BNL, CERN, DESY, SDSC,..... SuperSINET APAN/Transpac APAN

5. Since 2004.1 SuperSINET’s links to US/EU SuperSINET NII’s router at NewYork OC48 x 4 Gèant Abilene ESnet OC48 GbE ISP ( 10Gbps ) MAN LAN 10GbE Switch POS Channel 10GbE GbE x 3 IEEAF Atlantic SLAC is on ESnet CERN is on GEANT

6. WAN Performance Measurement and Monitor -- Study and Work at KEK Qi Fazhi 2004-07

7. Study WAN Performance Terminology WAN Performance Measurement Tools How to get larger throughput

8. Terminology / Elements be relationship to WAN Performance Bandwidth –Physical bandwidth, or capacity (C) –Available bandwidth (A) Throughput –Maximum throughput –Achievable throughput Latency / RTT

9. TERMINOLOGY Latency Packet Loss Bandwidth: the speed that a network element can forward traffic. (is independent of end hosts and protocol type) –Physical/capacity:maximum number of bits per second a network element can transfer(is determined by the slowest network element along the path) –Available:the capacity minus utilization over a given time interval. Throughput: amount of data that is successfully sent from one host to another via a network (may be limited by every component along the path from source host to destination host) –Maximum throughput –Achievable throughput

10. Network Performance Measurement Tools Iperf/Netperf: tools for measuring end-to-end TCP/UDP performance pipechar: hop-by-hop bottleneck analysis tool pchar: hop-by-hop performance measurement tool traceroute: lists all routers from current host to remote host

11. How to get larger throughput Decrease Latency / RTT Packet Loss Choose the best pathRouting coordination Larger Throughput TCP Tuning ? Latency / RTT Packet Loss Parallel stream TCP

12. TCP Tuning Why? –Bandwidth increases –TCP shortage How? –Modify the TCP Windows Size –Modify the System Buffer Size –…… –Some New Protocol: FAST

13. TCP TCP is adaptive It is constantly trying to go faster It slows down when it detects a loss How much it sends is controlled by windows When it sends is controlled by received ACK’s(or timeouts)

14. Summary of techniques to maximize TCP WAN throughput congestion window (CWND):TCP uses it to determine how many packets can be sent at one time, The larger the congestion window size, the higher the throughput. TCP “ slow start ” and “ congestion avoidance ” algorithms determine the size of the congestion window The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket (System) For each socket, there is a default value for the buffer size, which can be changed by the program using a system library call just before opening the socket. Kernel enforced maximum buffer size. Why Tuning?

15. Bandwidth*Delay Product Bandwidth * Delay = number of bytes in flight to fill path The optimal buffer size is twice the bandwidth*delay product of the link( recover from errors ) buffer size = 2 * bandwidth * delay = bandwidth * RTT bandwidth :get it from pipechar or pchar pipecharpchar RTT:get it from Ping Example:ping time is 50 ms ; the end-to-end network consists of all 100 BT Ethernet and OC3 (155 Mbps).then TCP buffers should be.05 sec * (100 Mbits / 8 bits) = 625 KBytes most of today UNIX OS's by default have a maximum TCP buffer size of only 256 KB! So we should do some tuning …… buffer Windows throughput BW*Delay

16. Important Points About TCP Throughput congestion window : determine how many packets can be sent at one time,the larger the congestion window size, the higher the throughput. the size of the congestion window : slow start ( constantly trying to go faster ) and congestion avoidance ( slows down when it detects a loss ) maximum congestion window : is related to the amount of buffer space that the kernel allocates for each socket buffer size : 2*Bandwidth*Delay Products （ BDP ） = bandwidth * RTT Summary

17. Routing coordination KEK IHEP Before Change Status –Go through Russia and USA KEK->IHEP: KEK->Sinet->Abilene->RBNet->CSTnet->IHEP IHEP->KEK: IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK –RTT is very large After Change Status –KEK->IHEP: KEK->SINet->APAN-JP->CERnet->NSFC->CSTnet->IHEP –IHEP->KEK (not changed): IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK –For the reason of not changing IHEP->KEK route,the RTT is still large,but there are some changes in the throughput

18. KEK-IHEP Route Tuning KEK2505 Sinet2907 Abilene 11537 APAN_JP 7660 RBnet5568 Dargon_tap 9407 Cernet 4538 NSFC 9406 CSTNet 7497 AS1239 AS9405 AS3356 Level 3 Sprint APAN_JP&APAN_CN Link IHEP3460

21. Parallel TCP Stream KEK->IHEP One TCP StreamParallel TCP Streams

22. WAN Performance Measurement/Monitor Target –Packet loss & latency –Throughput Tools Mechanism WORK

23. Test PC Data Remote WWW Traceroute/Ping iperf netperf HTTP Ping/iperf/netperf Mechanism (Base on www) CMDResult

24. System Structure WAN Performance Measurement and Monitor WAN Performance MeasurementWAN Performance Monitor Performance data collect Data File Performance data analysis Performance measurement resultPerformance measurement command

25. The System Interface

26. Measurement: (RTT & Packet Loss/traceroute & ping)

27. Traceroute and Ping Result

28. Traceroute and Ping Result with graph ms

29. Measurement: Throughput with Iperf

30. Measurement Result: Throughput with Iperf

31. Measurement: Throughput with Netperf

32. Measurement Result: Throughput with Netperf

33. Monitor: RTT and Packet loss with ping

34. Monitor Result: RTT and Packet loss with ping

35. Monitor: Throughput with iperf

36. Monitor Result: Throughput with iperf

37. README INSTALL –creat a user account: monitor –cd ~monitor –tar xvf monitor.tar –edit /usr/local/apache/conf/httpd.conf –edit the ~monitor/ping_list.txt and ~monitor/iperf_list.txt –edit the /etc/crontab –http://serveraddress/monitor/

38. Useful links http://www.apan.net http://ws.arin.net/cgi-bin/whois.pl