Networking for the Grid Yee-Ting Li eScience Summer Edinburgh

What the GRID is Worldwide Distributed System Worldwide Distributed System Interconnected with ‘networks’ Interconnected with ‘networks’ Balancing processors, storage and network utilization Balancing processors, storage and network utilization Networking is important to make GRID work Networking is important to make GRID work

Networking Important! Only way two grid nodes can communicate with each other Only way two grid nodes can communicate with each other Need ways of determining how ‘efficiently’ they talk Need ways of determining how ‘efficiently’ they talk Focus on: Focus on: –The characterising how they talk –The language they use to talk

Part 1 Networking Networking Networking Monitoring Networking Monitoring –Networks are also transient –Network performance also varies as you’re sharing with n million other users Sometimes you can notice periodic patterns – sometimes you can’t Sometimes you can notice periodic patterns – sometimes you can’t –Difficult to analyse and create trends/predictions –Show steps towards…

Networking 101 Networking straight forward Networking straight forward Just connect to the network and it works! Just connect to the network and it works! HA! HA!

Networking Complex? Get’s more complex! Complex? Get’s more complex! Each node has it’s own scheduling priorities Each node has it’s own scheduling priorities Routers must serve trillions of data units per second! Routers must serve trillions of data units per second!

Networking Complex stack from which data has to flow to get onto network Complex stack from which data has to flow to get onto network Each node on the network also has their own stacks Each node on the network also has their own stacks Routers have IPR on stacks – no one knows what Cisco stuff looks like! Routers have IPR on stacks – no one knows what Cisco stuff looks like!

Example Metrics Connectivity Connectivity Delay Delay –One-way delay –Two-way delay Throughput / goodput Throughput / goodput Network path Network path Loss Loss Jitter Jitter

Metrics Example Video Conferencing Video Conferencing –Needs predictable bit rate –Doesn’t usually matter if bit rate changes too much –Needs constant jitter –Low one-way delay preferable FTP FTP –Needs reliable transport –Throughput depends on urgency of data –Jitter and delay don’t matter

Network Monitoring Uses Monitoring is measuring over long periods of time Monitoring is measuring over long periods of time Gives an indication of network performance over time – a baseline Gives an indication of network performance over time – a baseline Allows comparison of different tools for analysis Allows comparison of different tools for analysis Allows analysis of how different protocols behave in different conditions – in real life Allows analysis of how different protocols behave in different conditions – in real life Allows ‘tuning’ of existing protocols to make most out of network Allows ‘tuning’ of existing protocols to make most out of network

Possible Users of a NM Web Service Network Managers Network Managers –See how much bandwidth is being used Network Analysts Network Analysts –Make things faster and better! Resource Brokers Resource Brokers Broker to determine where to send jobs – Network Cost Broker to determine where to send jobs – Network Cost Bandwidth Brokers Bandwidth Brokers –Allocate bandwidth depending on current network state Replication Managers Replication Managers –Distribute data only when network is not busy QoS Brokers (aka Managed bandwidth Services) QoS Brokers (aka Managed bandwidth Services) –Universal language for intercommunication..? Next Generation FTP Next Generation FTP –First look up historical throughputs before sending to determine best path

GridNM Architecture for monitoring the network Architecture for monitoring the network Backend – collects data for presentation Backend – collects data for presentation Logs metrics in ASCII log files on a single host Logs metrics in ASCII log files on a single host Allows mesh measurements – all nodes performs measurements to al other nodes Allows mesh measurements – all nodes performs measurements to al other nodes Uses standard UNIX infrastructure – ssh Uses standard UNIX infrastructure – ssh –Should be easily adaptable to using Globus certifications once interactive processing is introduced in EDG.

GridNM (cont…) Uses existing (and future tools) to collect metrics Uses existing (and future tools) to collect metrics Modular - uses XML to describe available resources Modular - uses XML to describe available resources –Hosts –Tools Locks hosts if under measurement – prevents other tests affecting metrics Locks hosts if under measurement – prevents other tests affecting metrics Currently monitoring 6 sites around Europe using 5 tools Currently monitoring 6 sites around Europe using 5 tools

GridNM ‘plot’

Web Service Network Monitoring GridNM just one Network Monitoring Program GridNM just one Network Monitoring Program Many different programs out there! Many different programs out there! Unify data exchange between different monitoring infrastructures Unify data exchange between different monitoring infrastructures

piPEs Internet2 e2ePI Architecture for network monitoring Internet2 e2ePI Architecture for network monitoring Defines information flow to diagnose networks and hosts performance – white paper Defines information flow to diagnose networks and hosts performance – white paper Incorporates a ‘finger pointing’ mechanism to identify poor performers Incorporates a ‘finger pointing’ mechanism to identify poor performers Ideal starting point! Ideal starting point! BUT… found out about it too late… BUT… found out about it too late… Currently investigating implementation with SLAC software + web service as possible implementation of piPEs software Currently investigating implementation with SLAC software + web service as possible implementation of piPEs software

GGF NMWG Defines characteristics that are just the values that we are interested in Defines characteristics that are just the values that we are interested in Defines classes of metrics, e.g. bandwidth, delay etc. that these characteristics report Defines classes of metrics, e.g. bandwidth, delay etc. that these characteristics report Defines singleton and derived characteristics Defines singleton and derived characteristics Defines samples of data and their inherent sampling patterns Defines samples of data and their inherent sampling patterns Timestamps Timestamps Still in draft form… Still in draft form…

GGF NMWG cont. / Schema Design As it’s all in XML, designing a XML schema to describe ‘objects’ to be passed around As it’s all in XML, designing a XML schema to describe ‘objects’ to be passed around XML Schema Document (XSD) XML Schema Document (XSD) Focusing actually implementing what the NMWG document says… and doesn’t say… Focusing actually implementing what the NMWG document says… and doesn’t say… Note: We are also tackling this from a pure OO design too – however, due to technical differences between objects in C++, Java and SOAP/XML then there may be issues to overcome… Note: We are also tackling this from a pure OO design too – however, due to technical differences between objects in C++, Java and SOAP/XML then there may be issues to overcome…

Part 2 Network Communication Languages Network Communication Languages Known as transport protocols - determines how applications put traffic into the network Known as transport protocols - determines how applications put traffic into the network Sits on top of IP – common language of the internet Sits on top of IP – common language of the internet

Transport Level Protocols TCP (HTTP, FTP, GridFTP) used for file transfer TCP (HTTP, FTP, GridFTP) used for file transfer –Gives guarantee on delivery –All data is copied precisely –Performance can be poor –Respects other internet users UDP (Real, H323) used for video conferencing UDP (Real, H323) used for video conferencing –Gives no guarantees on delivery –Data may be incomplete –Performance good –Doesn’t respect other internet users

UDP vs TCP Udp: min=274, max=565, ave=493, stdev=43 Udp: min=274, max=565, ave=493, stdev=43 Tcp: min=37, max=292, ave=195, stdev=40 Tcp: min=37, max=292, ave=195, stdev=40 Summary: tcp is rubbish! – why? Summary: tcp is rubbish! – why?

Memory and Disk transfers Iperf TCP Mbits/s File copy disk-to-disk Fast Ethernet OC3 Disk limited Over 60Mbits/s iperf >> file copy Les Cottrell, SLAC

What does TCP do? TCP retransmits lost data TCP retransmits lost data Even retransmits data it ‘thinks’ has been lost! Even retransmits data it ‘thinks’ has been lost! Needs and uses a ‘windowing’ system Needs and uses a ‘windowing’ system –Uses ACKnowledgements from reciever –Grows a Congestion Window ‘cwnd’ to determine the size of window Model: Model: –Tap is independent of Tank size –Tank filled by application –Valve opening (data rate) determined by feedback from network –Small tanks mean small data rate –Large tanks mean larger data rate Socket buffer size TCP Protocol Network

TCP socket buffer sizes Iperf observations: 490 Iperf observations: 490 Standard socket buffer graph Standard socket buffer graph –Shows linear(ish) region followed by plateau Optimal socket buffer size just over 2mB Optimal socket buffer size just over 2mB

Retransmitted Data Graph shows the amount of retransmitted data against the throughput Graph shows the amount of retransmitted data against the throughput Retransmitted data is due to loss on the network Retransmitted data is due to loss on the network General case ACK’s have to timeout before resending General case ACK’s have to timeout before resending We get more retransmitted data for low throughputs with large windows We get more retransmitted data for low throughputs with large windows

Measuring Performance of Transport Level Protocols Need to identify what we want to measure – the metrics. Need to identify what we want to measure – the metrics. Dependant on the use of the transport protocol. Need to analyse application level usage Dependant on the use of the transport protocol. Need to analyse application level usage For Grid: For Grid: –Movement of ‘transient’ data File Transfer and Replication File Transfer and Replication process jobs or ‘sandboxes’ process jobs or ‘sandboxes’ –Movement of Real-Time Data Video Conferencing – Access Grid Video Conferencing – Access Grid Real-Time applications Real-Time applications

Web 100 & TCP OSI states that we should not know anything about the separate layers OSI states that we should not know anything about the separate layers How do we know something is going wrong? – your throughput decreases! How do we know something is going wrong? – your throughput decreases! Prevents congestion collapse! Prevents congestion collapse! Need Web100! Allows in depth tcp stack analysis per flow Need Web100! Allows in depth tcp stack analysis per flow Kernel patch – , alpha1.2 Kernel patch – , alpha1.2 New version – alpha2.0pre1 New version – alpha2.0pre1 Using program to grab web100 results - logvars Using program to grab web100 results - logvars

Reliability of Web100 results… Still alpha… but reliable Still alpha… but reliable Graph against iperf throughputs correlate very well Graph against iperf throughputs correlate very well At least as reliable as the result offered by iperf! At least as reliable as the result offered by iperf!

Congestion Window Looking at the max_cwnd achieved for each measurement… Looking at the max_cwnd achieved for each measurement… Appears to be two regions Appears to be two regions –with high correlation of throughput and max cwnd –A linear region where we get the a range of throughputs for same max_cwnd Cwnd never grows beyond 1500kbytes! Cwnd never grows beyond 1500kbytes!

Bandwidth Delay Product Window = bandwidth * delay Window = bandwidth * delay We want We want –Bandwidth = 1,000,000,000 bit/sec We have We have –Delay = 19ms Window needs to be an average of… Window needs to be an average of… –=1e+9 * 19e-3 / 8 bytes –=2.25mbytes! We only achieve ~1.5mbytes max! We only achieve ~1.5mbytes max! Need to implement some monitoring of the degree of the average and variation of cwnd for each tcp connection… Need to implement some monitoring of the degree of the average and variation of cwnd for each tcp connection…

TCP Optimisation It’s actually TCP that is limiting our transfer rates! It’s actually TCP that is limiting our transfer rates! –All applications use it! Understandable as TCP hasn’t changed much for the last years! Understandable as TCP hasn’t changed much for the last years! –When standard link was about 56kbit/sec! Solution: Need new TCP implementations! Solution: Need new TCP implementations!

What is High Speed TCP? Changes the way TCP behaves at high speed (ie large cwnd) Changes the way TCP behaves at high speed (ie large cwnd) Standard TCP has two modes Standard TCP has two modes –Slow start (not very slow…) –Congestion Avoidance Focuses on Congestion Avoidance Region – ie when TCP knows (thinks it knows…) how well the network behaves… Focuses on Congestion Avoidance Region – ie when TCP knows (thinks it knows…) how well the network behaves… BUT only when we are at high speeds, else do what normal Standard TCP does… BUT only when we are at high speeds, else do what normal Standard TCP does… Readily deployable 1 st step towards Equation Based Congestion Control Readily deployable 1 st step towards Equation Based Congestion Control

What does it do? Standard TCP uses two parameters Standard TCP uses two parameters –Increase parameter, a –Decrease parameter, b i.e. AIMD( a,b ) i.e. AIMD( a,b ) Standard TCP uses Standard TCP uses –a=1 –b=0.5 High Speed TCP introduces High Speed TCP introduces –a->a(cwnd) –b->b(cwnd) i.e. The value of a and b depends on the current congestion window size i.e. The value of a and b depends on the current congestion window size If we increase a more with larger cwnd we can get back up to our ‘optimal’ cwnd size for the network path If we increase a more with larger cwnd we can get back up to our ‘optimal’ cwnd size for the network path If we decrease b less we don’t lose as much bandwidth due to a small congestion window If we decrease b less we don’t lose as much bandwidth due to a small congestion window

What exactly does it do? Based on the TCP response function Based on the TCP response function –Relates loss and throughput Uses the TCP response function to investigate certain parameters Uses the TCP response function to investigate certain parameters –High_Window, High_Loss; largest cwnd needed for x throughput and the required loss for that throughput –Low_Window, Low_Loss; smallest cwnd when we actually switch from Standard TCP and the required loss rate for that cwnd size –High_B; the smallest decrease in b when we are at a large cwnd Equations to transform this information into a table for a(cwnd) and b(cwnd) Equations to transform this information into a table for a(cwnd) and b(cwnd)

Transport Protocols ‘NG’ NameTransportNotes UDP BlastUDP TsunamiUDP/TCPUses TCP as ‘control’ channel High Speed TCPTCPFor 10Gb/sec links PGM / CCModified UDPMulticast UDP – new transport protocol IBPApplication ‘logistical networking’