2. First of ALL Big appologize for Kei’s absence

3. Hero of this year’s LSR achievement Takeshi in his experiment

4. What is Data Reservoir? Share Scientific Data over long distance –Physics, astronomy, earth science, biology High-speed data transfer on Long Fat pipe Network Easy to use –File system transparent

5. Data Reservoir System User Programs Disk Server IP Switch File Server Disk Server IP Switch Disk Server iSCSI Bulk Transfer Global Network Using iSCSI protocol Without any modification on applicatoins

6. 3 rd Generation SC04 SC05 Round the World 31,248km 1 to 1, memory to memory transfer Single Stream, Longest Path, Standard MTU TCP Throughput Award Fastest IPv6 Hisotry of Data Reservoir and SC BandWidth Challente 1 st Generation SC02 26 to 26 servers 1GbE interface RTT 200ms, 90 ％ usage of bottleneck OC-12 2 nd Generation SC03 Aggregated 10Gbps 24,000Km 1 and a half round trip between U.S. Tokyo 32 to 32 Servers too many :-< 4 th Generation SC06 A pair of machines Disk to Disk transfer Single 7.2Gbps Dual 8.65 Gbps

7. Once upon a time, There started an ambitious project to construct an L2 network between CERN and Tokyo via Amsterdam, Canada, and U.S. Fortunately ( ! ), our team got a chance to try it ♪

8. Network Tokyo CERN Pittsburgh Chicago Amsterdam Geneva Seattle Vancouver Calgary Minneapolis WIDE APAN/JGN II IEEAF/Tyco/WIDE CANARIE SURFnet Abilene

9. 3 rd Generation Data Reservoir started Background WAN PHY over the world Programmable 10GbE NIC is available Challenge How much bandwidth can we use by single stream?

10. Struggles while the 1 st experiment Almost no information –Ping + loopback is the only source –Different network, different timezone –TELEPHONE must be the most important equipment. Over 7Gbps between Tokyo and CERN

11. It is nice of this experiment to have a lot of new friend! We really appreciate nice adivces. Submission to Internet2 Land Speed Record Experiments while X’mas vacation, the smallest traffic season!

12. Some Results SC04 Band Width Challenge U.S. – Tokyo – U.S. – CERN 31,248km, RTT 433ms, 7.57Gbps Xmas Experiment Season with smallest network traffic. Very Very strict dead-line for preparation Tokyo Chicago Amsterdam Siattle Tokyo 33,979km, RTT 498ms 7.21Gbps : Update LSR 8times.

13. Network Tokyo CERN Pittsburgh Chicago Amsterdam Geneva Seattle Vancouver Calgary Minneapolis WIDE APAN/JGN II IEEAF/Tyco/WIDE CANARIE SURFnet Abilene

14. Challenge in 2006 To attain 90% of 10Gbps The difficulty WAN PHY (MAX 9.6Gbps) ⇔ LAN PHY Only 4% of 10Gbps, But, if RTT = 500, the difference is 25MBytes for Round Trip (TCP can control transmission rate with RTT grain) Another difficulty PCI-X bottleneck → Now, cleared

15. LSR in 2006 -- New players Circuit -- NetIron 40G  NetIron RX-4 in Seattle GSO (Generic Segmentation Offload ) –Offloading CRC calculation Chelsio T310 -- PCI-X2.0 support IPG tuning is available Iperf modification with sendfile() Hardware Approach for 10Gbit Network TAPEE: Network Analyzer

16. 2006 LSR Challenge, again on X’mas Around Dec/10: Seattle line test Around Dec/20: Round-The-World up Dec/31: Submission Jan/8/2007: Round-The-World down

17. Host Xeon 5160 * 1 –Woodcrest core –Dual core DDR400 2GB Chelsio T310-SR on PCI-Express x8 –There is no longer bus speed bottleneck Linux 2.6.18

18. Circuit Round The World circuit –522ms RTT –Trans Pacific & Trans Atlantic –WAN PHY & LAN PHY mixed –Tokyo – [Los Angels] – Chicago – Amsterdam –Amsterdam – [Chicago] – Seattle – Tokyo

19. Amsterdam NetherLight At SARA SURFnet IEEAFCANARIE L3 switch Chicago StarLight L2 switch Atlantic Ocean Pacific Ocean WAN PHY Force10 E1200 HDXc Foundry RX-4 Seattle Pacific Northwest Gigapop SURFnet WIDE JGN2 ONS 15454 ONS 15454 Foundry NI40G GS4000 WAN PHY HDXc GS4000 Others L1 switch T-LEX IEEAF WAN PHY LAN PHY JGN2 LAN PHY CANARIE CA* NET 4 WIDE WAN PHY LAN PHY LSR 200612-2 Network Topology Foundry RX-4 WAN PHY Age-1 Intel Xeon Age-2 Intel Xeon Fujitsu XG800 JGN2 Tokyo Force10 E300 JGN2 Los Angels JGN2 WAN PHY CISCO 7609 HDXc SURFnet WAN PHY NYC MANLAN TransLight LAN PHY TransLight

20. LSR distance From To Distance HND (35°33'08"N 139°46'47"E) ORD (41°58'43"N 87°54'17"W) 10147 km ORD (41°58'43"N 87°54'17"W) AMS (52°18'31"N 04°45'50"E) 6630 km AMS (52°18'31"N 04°45'50"E) SEA (47°26'56"N 122°18'34"W) 7864 km SEA (47°26'56"N 122°18'34"W) HND (35°33'08"N 139°46'47"E) 7730 km 4 segment path: 32372 km

21. IPG Tuning Chelsio T310 has special function of setting IPG (Inter Packet Gap) –Enables to control the Ethernet NIC transmission rate –Upto 2048 octet (IEEE standard IPG 12 octet) Fine Grain Tuning For Standard Frame control 50 ～ 100 %, For 8000B Jumbo Frame 80 ～ 100%

22. Without pacing (IPG 136) 600MB RWIN

23. Pacing (IPG 800) 600MB RWIN

24. Pacing (IPG 700) 600MB RWIN

25. Pacing (IPG 720) 600MB RWIN

26. Iperf modification We have been used Iperf Iperf transmission flow –Allocate several kB buffer –Initialize buffer with random data –while() { write(sock, buffer) } This invokes copy between user and kernel space

27. Iperf modification (cont’d) An advice from Chelsio –“Use netperf’s sendfile mode to confirm receiver performance” Modification –Iperf-zerocopy transmission flow open(temporary file)  file descriptor fd buffer = mmap(fd) initialize buffer with random data while() { sendfile(sock, fd) } –sendfile(2) sends data from kernel After some discussion, we concluded that using this version of Iperf meets LSR rule

28. GSO

29. GSO + zerocopy

30. New submission 7.67Gbps average –Standard-Iperf –Peak 8.10Gbps, 20 minutes, No packet loss 9.08Gbps average –Iperf-zerocopy –Peak 9.11Gbps, 5 hours, No packet loss

31. History of single-stream IPv4 Land Speed Record 2000200120032004200520062007 Year 1 10 100 Distance bandwidth product Pbit m / s 2004/11/9 Data Reservoir project WIDE project 149 Pbit m / s 2002 1,000 2005/11/10 240 Pbit m / s 10 Gbps * 30,000km 2006/2/20 264 Pbit m / s 2004/12/24 216 Pbit m / s

32. History of single-stream IPv6 Land Speed Record 2000200120032004200520062007 Year 1 10 100 Distance bandwidth product Pbit m / s 2004/10/29 Data Reservoir project WIDE project 167 Pbit m / s 2002 1,000 2005/11/13 Data Reservoir project WIDE project 208 Pbit m / s 10 Gbps * 30,000km 2006/12/28 Data Reservoir project WIDE project 272 Pbit m / s