1. Bio-Informatics Workshop, Beijing , October 2009
GÉANT: European & World Connectivity Network Performance & Data Transfers Linking EBI & China
Richard Hughes-Jones
DANTE Delivery of Advanced Network Technology to Europe
Bio-Informatics Workshop, Beijing , October 2009

2. European R&E Networking
Each country operates it own independent R&E network.
Depending on the country, this can connect universities, colleges, research laboratories, schools, libraries, museums, municipal offices, hospitals.
These national research and education networks (NRENs) may use different technologies and offer different services.
The NRENs are interconnected by a pan-European backbone called GÉANT.
GÉANT provides extensive international connectivity to other world regions and is operated by DANTE.
The key is close Collaboration & Co-operation.

3. The GÉANT – NREN Evolution:
7th generation of pan-European research network federated infrastructure: A 20 year success story
EuropaNET  TEN34  TEN155  GÉANT (GN1 GN2  GN3)
Connects 36 European countries through 32 NREN partners.
Serves over 3,500 research and education establishments across Europe
Over 30 million users
Provides extensive international connectivity to other world regions.
Funded jointly by NRENs and the EC.
Mechanism is the GN3 Collaborative Project:
co-ordination by DANTE via the PMT (Project Management Team)
complemented by TERENA
involves > 400 NREN staff
Collaboration & Co-operation.
Vasilis Maglaris

4. GÉANT GN3 Objectives Provide a dependable Network Expand the reach
Complete the GÉANT European Footprint
New partners in South Eastern Europe
Associate and Cooperate with Eastern Europe partners
Widen the user base
e-Infrastructure (ESFRI) projects including :
Earth Observation,
Climatology,
Space and Fusion,
Bio & Life Sciences,
Arts and Humanities

5. 38 European Countries Dark Fibre Hybrid network: IP Packet routed
Switched point-to-point Circuits
Dedicated wavelengths Lambdas
25 POPs (+4) serve >30 NRENs
11600 km of fibre ILA sites
50+ x (own) 10G lambdas
9 x (leased) 10G lambda
8 x 2.5G (leased) “lambda” + some lower speed links
Juniper T640, M160, M40 routers
NREN accesses at up to 10Gbps (+ backup) + P2P
GÉANT topology – April 2009

6. Network Connectivity Services
GÉANT & NRENs will offer connectivity. Hybrid networks built on dark fibre infrastructures lit with DWDM equipment.
Basic IP access via a GÉANT2 router - GÉANT ip
High Bandwidth
Multicast
IPV6
QoS
MPLS / Traffic engineering
Point-to-point Circuit Network - GÉANT plus
Dedicated Capacity – typically 1 Gigabit paths
Quickly configurable
More than 60 dedicated circuits by January 2009
Wavelengths - GÉANT lambda
Full 10Gbps wavelengths across Europe – linking data centres
Note that not all networks may be able to offer all these facilities.
Key phrase is Multi-Domain

7. Network Services GÉANT & NRENs will offer network services.
Network Monitoring
perfSONAR - based on OGF Standards
e2eMON
(Troubleshooting tool EGEE)
eduPERT
Network Provisioning “on demand”: Circuits / VPNs / MPLS / QoS
autoBAHN tool from GN2
World wide interoperation – OGF Standard being developed
Quick configuration tool as a minimum
Security & mobility
PKI coordination
Roaming Access Service eduROAM
eduGAIN Service introduction
Note that not all networks may be able to offer all these service facilities.
Key phrase is still Multi-Domain

9. The Hybrid GÉANT2+ services
Lambdas
Lambdas
Primary IP
Peering IP
Backup
GÉANT+ IP
Backup
Primary IP
Peering
GÉANT+
To NREN A
To NREN B
IP Router
Switch
Switch
IP Router
Backup IP
Peering for
NREN A
Dark fibre/amplifier chain
GÉANT
POP A
GÉANT
POP B

10. GÉANT global connectivity
Includes EU Developmental Initiatives

11. EUMEDCONNECT Topology April 2008
Global Connectivity
ALICE Project: RedCLARA Topology October 2007
TEIN3 Topology January 2009
EUMEDCONNECT Topology April 2008

12. TEIN3 Topology January 2009 TEIN2 2005 – 2008 Now TEIN3
Federation of NRENs
Upgrade:
Smaller links 64Mbit 115Mbit
Larger link 622Mbit  2.5 Gbit
2.5 Gbit Singapore to Europe
SG & HK routers M 120
NOC in HC
Short RTTs

13. Network Monitoring

14. perfSONAR: Using NM-WG Schema
perfSONAR uses the OGF NM-WG Recommendations (ontology) and Schemata for the API
OGF NMC-WG for protocols
perfSONAR is performance middleware
Modular
Web services-based
Decentralized
Integrates:
Network measurement tools
Network measurement archives
Discovery
Authentication and authorization
Data Manipulation
Topology
Visualization 14

15. perfSONAR Uptake & Deployment
perfSONAR is a joint effort:
ESnet
GÉANT2 JRA1
Internet2
RNP (Brazil)
ESnet includes:
ESnet/LBL staff
Fermilab
Internet2 includes:
University of Delaware
Georgia Tech
SLAC
Internet2 staff
GÉANT2 JRA1 includes:
Arnes
Belnet
Carnet
Cesnet
CYNet
DANTE
DFN
FCCN
GRNet
GARR
ISTF
PSNC
Nordunet (Uninett)
Renater
RedIRIS
Surfnet
SWITCH
Other contributions & installations:
CLARA (Latin American Cooperation of Advanced Networks)
LHC Network
JANET
An OGF success story: 27 partners 15

16. Weather-map E2Emon Link Status

17. Projects Using GÉANT

18. Sichuan Earthquake Recovery Efforts
Sichuan Earthquake on 12th May 2008
Satellite images of disaster region transferred over the ORIENT link from JRC (Italy) to RSGS (China) with routed IP.
Tuned TCP throughput
Added a second GE link from CSTnet in China.

19. VLBI 1GE Circuits & IP connections in 2008
1 Gigabit Ethernet point to point circuits to the telescopes in Europe.
Supplied by NRENs & GÉANT
Also use Routed IP to other world regions
Timeliness very important
Data rates: 64Mbit/s 512 Mbit/s 1 Gbit

20. GÉANT lambdas form the DEISA Network 2008
All the Super Computers use 10 G Ethernet links connect to a 10 GE switch in DFN Frankfurt.
Also use the routed IP service.

21. GÉANT lambdas form the LHC Optical Private Network
10 Gigabit Lambdas form a dual star on 2 routers at CERN.
Connects National data centres to CERN
DANTE advised on architecture & resiliency.
Have backup links.
LHC operate the OPN
Use Routed IP for other international traffic
Edoardo Martelli
LHC

22. Collaborative projects with high societal impact: eHealth & eLearning
WISDOM:
ICT-LEAP (EC-TEMPUS project)
Massive biomedical data challenge
Involves parallel use 5000 computers in 27 countries (~ 420 years’ computing for a single PC)
e-Infrastructure accelerate screening of drugs against malaria, avian flu and other emerging diseases
Links universities from Polar Area to Middle East
Facilitates e-learning centres in Middle East
Broadens access to education
Virtual enrolment in remote lectures
Virtual capacity building in e-learning of teaching staff

23. Working with Emerging Users
GEO / GEOSS
Accepted as a participating Organisation.
Areas: BW for disasters, linking data centres, Africa FEAST,GEONET
EUMETSAT
Discussed requirements.
working with them on options for using the academic network.
Collaborative tests on high BW multicast.
Bio-Informatics
Examine current connectivity.
Interest in creating a Bio-NREN forum.
ITER
Require high bandwidth data transfers.
Assisting multi-gigabit transfers to Japan.

24. Current & Emerging Users
China – Europe
Current & Emerging Users
Particle Physics iHEP
Radio Astronomy eVLBI
"World Lecture Theatre" with Shanghai Jiao Tong University (SJTU)
GEO Earth Observations GEONETtCast
Deutscher Wetterdienst
EUMETSAT - China Meteorological Administration
Bio-Informatics
ITER - Fusion
Grids - INWA-GRID, Bridge-Grid

25. Network Path Performance

26. Beijing – Stockholm UDP Throughput for eVLBI
In collaboration with the Radio Astronomers Sep 2008:
We made measurements of throughput, packet loss, jitter to our test PC in the PoP in Beijing & to VLBI PCs.
VLBI data requires UDP flows of 512Mbit/s
Provide liaison with the Chinese NRENs CSTnet & CERNET

27. ESLEA and UKLight at SC|05
Disk-to-disk transfers with bbcp
Seattle to UK
Set TCP buffer / application to give ~850Mbit/s
One stream of data 840 Mbit/s
Stream UDP VLBI data
UK to Seattle
620 Mbit/s
No packet loss
Worked well

28. Help with TCP Throughput
BaBar on Point2Point Circuit
HighSpeed TCP
Linespeed: 940 Mbit/s
DupACKs <~10 (expect ~400 for loss)
BaBar on routed IP network
Standard TCP
425 Mbit/s
DupACKs – re-transmits
Packet loss is the killer 28

29. Network Path Performance use cases

30. LHC ATLAS Remote Computing Concepts
Remote Event Processing Farms
Copenhagen
Edmonton
Krakow
Manchester
~PByte/sec
ATLAS Detectors – Level 1 Trigger
ROB
ROB
ROB
ROB PF PF
Data Collection Network PF PF
Event Builders
lightpaths
SFI
SFI
SFI
L2PU
Back End Network
L2PU
L2PU
L2PU
Level 2 Trigger
GÉANT
Local Event Processing Farms PF
SFOs
CERN site
320 MByte/sec
Switch PF
Experimental Area
Mass storage

31. ATLAS Remote Computing: Application Protocol
Send OK
Send event data
Request event
●●●
Request Buffer
Send processed event
Process event
Time
Request-Response time (Histogram)
Event Filter Daemon EFD
SFI and SFO
Event Request
EFD requests an event from SFI
SFI replies with the event ~2Mbytes
Processing of event
Return of computation
EF asks SFO for buffer space
SFO sends OK
EF transfers results of the computation
tcpmon - instrumented TCP request-response program emulates the Event Filter EFD to SFI communication.

32. TCPmon: TCP Activity Manc-CERN Req-Resp
Round trip time 20 ms
64 byte Request green 1 Mbyte Response blue
TCP in slow start
1st event takes 19 rtt or ~ 380 ms
TCP Congestion window gets re-set on each Request
TCP stack RFC 2581 & RFC reduction of Cwnd after inactivity
Even after 10s, each response takes 13 rtt or ~260 ms
Transfer achievable throughput 120 Mbit/s
Event rate very low
Application not happy!

33. TCPmon no TCP cwnd moderation: TCP Activity Manc-CERN Req-Resp
Round trip time 20 ms
64 byte Request green 1 Mbyte Response blue
TCP starts in slow start
1st event takes 19 rtt or ~ 380 ms
TCP Congestion window grows nicely
Response takes 2 rtt after ~1.5s
Rate ~10/s (with 50ms wait)
Transfer achievable throughput grows to 800 Mbit/s
Data transferred WHEN the application requires the data
3 Round Trips
2 Round Trips

34. TCPmon no TCP cwnd moderation: TCP Activity Alberta-CERN Req-Resp
Round trip time 150 ms
64 byte Request green 1 Mbyte Response blue
TCP starts in slow start
1st event takes 11 rtt or ~ 1.67 s
TCP Congestion window in slow start to ~1.8s then congestion avoidance
Response in 2 rtt ,300ms after ~2.5s
Rate 2.2/s (with 50ms wait)
Transfer achievable throughput grows slowly from 250 to 800 Mbit/s

35. Tests: Europe to China

36. Packet Loss & TCP

37. How it works: TCP Slowstart
Probe the network - get a rough estimate of the optimal congestion window size
The larger the window size, the higher the throughput
Throughput = Window size / Round-trip Time
exponentially increase the congestion window size until a packet is lost
cwnd initially 1 MTU then increased by 1 MTU for each ACK received
Send 1st packet get 1 ACK increase cwnd to 2
Send 2 packets get 2 ACKs increase cwnd to 4
Time to reach cwnd size W TW= RTT*log2 (W) (not exactly slow!)
Rate doubles each RTT
CWND
slow start: exponential increase
congestion avoidance: linear increase
packet loss
time
retransmit: slow start again
timeout

38. TCP (Reno) – Recovery Time
What Time for TCP to recover its throughput from 1 lost byte packet given by:
For rtt of ~200 1 Gbit/s:
2 min
UK 6 ms Europe 25 ms USA 150 ms China 200ms 1.6 s s min min

39. Packet Loss and new TCP Stacks
TCP Response Function
UKLight London-Chicago-London rtt 177 ms
2.6.6 Kernel
Agreement with theory good
Some new stacks good at high loss rates

40. Network Topology EBI, London, Beijing & Stockholm
Multi:
Vendor
Technology
Domain

41. UDP Throughput & loss ah04 – Beijing
Coalescence ON
MTU 1500 bytes
Max throughput 928 Mbit/s
Packet loss ~60% for small packets ≤ 200 bytes & < 5µs
These losses are mainly in network
1-10% losses (sometimes in end host)

42. TCP reno Throughput Buffer Scan ah04 – Beijing
Traceroute: PC-hinxton-hinx-lonJ-lon-ams-cop-BJ-PC
RTT 203 ms
BDP TCP buffer 25 MByte
Mathis (1997) TCP reno
Get about 2 Mbit/s for 0.1% packet loss
Some understanding to be done !

43. UDP Throughput & loss ah04 – London
Londonah04
0.1% packet loss for 1472 byte at line speed

44. TCP reno Throughput Buffer Scan ah04 – London
Traceroute: PC-hinxton-hinx-lonJ-lon-PC
RTT 2.6 ms
BDP TCP buffer 325 kByte
EBIlon as expected UDP is OK here too.
Lon EBI ??
Mathis (1997) TCP reno
Expect about 140 Mbit/s for 0.1% packet loss

45. TCP reno Throughput web100 London  ah04
TCP buffer 200k bytes
TCP Congestion Window opens OK
Many dupACKs with time-outs
Packets re-transmitted
iperf Ave throughput 565 Mbit/s (higher at start)

46. TCP reno Throughput web100 London  ah04
TCP buffer 300k bytes – bursts larger
TCP Congestion Window opens OK
Many time-outs and losses
Packets re-transmitted
iperf Ave throughput 580 Mbit/s

47. TCP bic Throughput web100 London  ah04
TCP buffer 200k bytes
TCP Congestion Window opens OK
Many time-outs
Packets re-transmitted
iperf Ave throughput 700 Mbit/s

48. UDP Throughput & loss London – Beijing
0.1% packet loss for large packets at line speed
> 60% loss <400 bytes & <8 us
Beijing  London
1-7 % packet loss for large packets at line speed

49. TCP reno Throughput Buffer Scan London – Beijing
Traceroute: PC-lon-ams-cop-BJ-PC
RTT 201 ms
BDP TCP buffer 25 MByte
BJ lon ~140 Mbit
LonBJ disappointing

50. TCP reno Throughput web100 London  Beijing
TCP reno buffer 25M bytes
TCP Congestion Window classic ,
Two packets re-transmitted at the start
iperf Ave throughput 40 Mbit/s
TCP bic buffer 15M bytes
Several timeouts
Many packets re-transmitted
iperf Ave throughput 115 Mbit/s

51. UDP Throughput & loss Stockholm – Beijing
Max throughput 926 Mbit/s
0.3% packet loss for large packets at line speed
> 60% loss <400 bytes & <8 us
London Beijing
0.1% packet loss for large packets at line speed
> 60% loss <400 bytes & <8 us

52. TCP reno Throughput Buffer Scan Stockholm – Beijing
Traceroute: PC-sto-cop-BJ-PC
RTT 201 ms
BDP TCP buffer 25 MByte
Plots smoother
BJ Sto ~150 Mbit similar to BJ-lon
Sto  BJ ~ 200 Mbit

53. TCP reno Throughput web100 Stockholm  Beijing
Send-stall
TCP reno buffer 25M bytes
TCP Congestion Window classic , send-stall
No packets re-transmitted
iperf Ave throughput 200 Mbit/s
TCP bic buffer 15M bytes
TCP Congestion Window + send-stall
No packets re-transmitted
iperf Ave throughput 700 Mbit/s

54. China – Europe Some next steps
The following is input to our discussions.
Complete the understanding of network performance & transfer rates
Fully involve NRENs & sites in end-to-end tests
Need to identify user projects & set the routing to allow use of the ORIENT link
Measure & estimate future load on the access links to TEIN3 & ORIENT

55. Any Questions ?

56. Backup Slides

57. GÉANT Links to the US All 10 Gigabit (or OC192) Route Funding Usage
MAX GigaPoP – Frankfurt GÉANT IP SDH (Washington mid Atlantic Crossroads)
MANLAN – Amsterdam IRNC/NSF IP SDH
New York – Paris GÉANT circuits 10GE/SDH
New York – London Internet2 circuits 10GE/SDH

58. Fibre Footprint 2004 Fibre/Wavelength Footprint GÉANT2+NRENs 2004
Black = Leased 10 Gbps wavelengths
Red=Fibre carrying 10Gbps wavelengths
Black: Leased Lines
Red: Dark Fibre

59. Fibre Footprint 2008 Fibre/Wavelength Footprint GÉANT2+NRENs 2008
Black = Leased 10 Gbps wavelengths
Red=Fibre carrying 10Gbps wavelengths
Black: Leased 10Gigabit
Red: Dark Fibre

60. The ALICE Project
The ALICE Project EU Development Initiative RedCLARA Network Topology, October 2007

61. RedCLARA
RedCLARA Network Topology 2009

62. TCP Flow Control: Receiver – Lost Data
Received but not ACKed
ACKed but not given to user
Window slides
Lost data
Data given to application
Last ACK given
Next byte expected
Expected sequence no.
Receiver’s advertised
window advances
leading edge
Application reads here
If new data is received with a sequence number ≠ next byte expected Duplicate ACK is send with the expected sequence number

63. Packet Loss with new TCP Stacks
TCP Response Function
Throughput vs Loss Rate – further to right: faster recovery
Drop packets in kernel
Managed Bandwidth
MB-NG
MB-NG rtt 6ms DataTAG rtt 120 ms

64. VLBI Application Protocol: The Effect of Packet Loss
Data1
●●●
Timestamp1
Time
TCP & Network
Receiver
Timestamp2
Sender
Data2
Timestamp4
Timestamp5
Data4
Timestamp3
Data3
Packet loss
VLBI data is Constant Bit Rate
tcpdelay
instrumented TCP program emulates sending CBR Data.
Records relative 1-way transit times of the data

65. Constant Bit-Rate over TCP
When there is packet loss
TCP decreases the rate.
TCP buffer 0.9 MB (BDP)
RTT 15.2 ms
Effect of loss rate on message arrival time.
TCP buffer 1.8 MB (BDP) RTT 27 ms
Timely arrival of data
Can TCP deliver the data on time?

66. Constant Bit-Rate over TCP
Very large TCP buffer: 160 MB
RTT 15.2 ms
Drop 1 in 1.12 million packets
Data Rate: 525 Mbit/s
Peak throughput ~ 734 Mbit/s
Min. throughput ~ 252 Mbit/s
TCP does regain timely delivery
Peak delay also large: 2.5 s

67. TCP Flow Control: Sender – Congestion Window
Uses Congestion window, cwnd, a sliding window to control the data flow
Byte count giving highest byte that can be sent with out an ACK
Transmit buffer size and Advertised Receive buffer size important.
ACK gives next sequence no to receive AND The available space in the receive buffer
Timer kept for each packet
Unsent Data
may be transmitted immediately
Sent Data
buffered waiting ACK
TCP Cwnd slides
Data to be sent,
waiting for window
to open.
Application writes here
Data sent and ACKed
Sending host
advances marker
as data transmitted
Received ACK
advances trailing edge
Receiver’s advertised
window advances
leading edge

68. TCP Congestion Avoidance
How it works:
TCP Congestion Avoidance
additive increase: starting from the rough estimate, linearly increase the congestion window size to probe for additional available bandwidth
cwnd increased by 1 segment per rtt
cwnd increased by 1 /cwnd for each ACK – linear increase in rate
TCP takes packet loss as indication of congestion !
multiplicative decrease: cut the congestion window size aggressively if a packet is lost
Standard TCP reduces cwnd by 0.5
Slow start to Congestion Avoidance transition determined by ssthresh
CWND
slow start: exponential increase
congestion avoidance: linear increase
packet loss
time
retransmit: slow start again
timeout