GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 1 Networking in Under 30 Minutes ! Richard Hughes-Jones, University of Manchester

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 2 E-science core project MB - NG u Project to investigate and pilot:  end-to-end traffic engineering and management over multiple administrative domains – MPLS in core diffserv at the edges.  Managed bandwidth and Quality-of-Service provision. (Robin T)  High performance high bandwidth data transfers. (Richard HJ)  Demonstrate end-to-end network services to CERN using Dante EU-DataGrid and to the US DataTAG. u Partners:CISCO, CLRC, Manchester, UCL, UKERNA plus Lancaster and Southampton (IPv6) u Status:  Project is running with people in post at Manchester and UCL.  Project Tasks have been defined and Detailed planning in progress  Kit list for the routers given to Cisco  Test PC ordered  UKERNA organising core network and access links – SJ4 10Gbit upgrade  Strong Links with GGF

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 3 MB – NG SuperJANET4 Development Network (22 Mar 02) Gigabit Ethernet 2.5 Gbit POS Access 2.5 Gbit POS core MPLS Admin. Domains Dark Fiber (SSE) POS WorldCom SuperJANET4 Production Network Leeds SJ4 Dev C-PoP Warrington SJ4 Dev C-PoP London UCL OSM-4GE- WAN-GBIC OSM- 1OC48- POS-SS UCL OSM-4GE- WAN-GBIC MCC OSM-4GE- WAN-GBIC OSM- 1OC48- POS-SS MAN OSM-4GE- WAN-GBIC RAL OSM-4GE- WAN-GBIC OSM- 1OC48- POS-SS RAL OSM-4GE- WAN-GBIC OC48/POS- SR-SC SJ4 Dev ULCC OC48/POS- SR-SC WorldCom SJ4 Dev C-PoP Reading OC48/POS- LR-SC WorldCom ULCC MB - NG

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 4 Defining Background Traffic u Regular traffic - constant size packet, regular spaced in time u Poisson traffic - constant size, exponential spacing to form transient queues u IETF traffic mix - different sizes and different probability of each size sent u Play back of real traffic patterns generated from packet headers pre-recorded from suitable points of the production network. This might include:  Video Conference traffic -> play back - rude/crude tools  UCL real conf playback tool  General traffic captured at edge of a site, e.g. Manchester  Do tests with a gen to see what gets dropped 0.5 Gbit typical peak UCL u Web-bursty traffic – web mirror – wget u Need to be able to reproduce traffic Statisticaly In general UDP best to understand the net Consider UDP TCP flows u Need ToS / QoS to be set u How to control  Start Stop  measure load as function of time – links and Queues  Start and end numbers

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 5 Defining the Measurements (10) u UDP round-trip latency vs packet size & Histograms  Sum of dt/dl transfer rates  Router & stack performance  Indication of network load & Switching / transport effects – spikes u UDP 1-way delay u UDP throughput vs Packet size and transmit delay  Throughput behaviour  Offered vs achieved throughput u UDP Packet Loss vs transmit rate and burst size  Loss rate Packet loss distribution as function of time  Buffer sizes in the path & Detect packet re-ordering u UDP Inter-frame Jitter as function of packet transmit spacing  Indication of network load  Behaviour of end system NICs u TCP round-trip latency vs Message size & Histograms  Sum of dt/dl transfer rates  Stack / protocol performance – detect Packet size dependencies u TCP throughput vs Message size and transmit delay  Throughput behaviour cf UDP  Packet loss distribution as function of time + Re-transmit rate u TCP throughput vs Window size / TCP tuning u TCP throughput vs number of streams  Stream throughput – benefits & effect on the network  Packet loss distribution as function of time + Re-transmit rate u TCP Protocol behaviour - tcptrace Align Metrics with GGF/IETF

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 6 Defining the Measurements (11) u TCP round-trip latency vs Message size  Sum of dt/dl transfer rates  (Router performance)  Stack / protocol performance  Detect Packet size dependencies u TCP round-trip histograms  Indication of network load  (Switching / transport effects – spikes)  Stack / protocol performance u TCP throughput vs Message size and transmit delay  Throughput behaviour cf UDP  Offered vs achieved throughput  Packet loss distribution as function of time + Re-transmit rate  Loss as func of pkt rate eg keep the data rate the same change pkt size – multi-streams u TCP throughput vs Window size / TCP tuning u TCP throughput vs number of streams  Stream throughput - benefits  Packet loss distribution as function of time + Re-transmit rate  Effect on Network u TCP Protocol behaviour - tcptrace  What are the “burst” lengths  Effect of routers / end system NICs u All this for WRED Wt fair Qing data rate const and ch pkt size – chack how well the routers do the Qing Align Metrics with GGF/IETF

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 7 The EU DataTAG project u EU Transatlantic Girid project. u Status: Well under way – People in post, Link expected Jul 02 u Partners: CERN/PPARC/INFN/UvA. IN2P3 sub-contractor u US Partners: Caletch, ESnet, Abilene, PPDG, iVDGL … u The main foci are:  Grid Network Research including: Provisioning (CERN) Investigations of high performance data transport (PPARC) End-to-end inter-domain QoS + BW / network resource reservation Bulk data transfer and monitoring (UvA)  Interoperability between Grids in Europe and the US PPDG, GriPhyN, DTF, iVDGL (USA) PPDG, GriPhyN, DTF, iVDGL (USA)

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 8 DataTAG Possible Configuration multi-platform multi-vendor Abilene ESNET UK SuperJANET4 IT GARR- B IT GARR- B GEANT NL SURFne t NL SURFne t 2.5 Gbit PoS lambda SLAC Cisco Juniper Alcatel Cisco Juniper Alcatel Giga Switch Juniper Light Switch Giga Switch Cisco 6509 Juniper Light Switch CERN (Geneva) Starlight (Chicago) Fermi

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 9 The SuperMicro P4DP6 Motherboard uDual Xeon Prestonia (2cpu/die) u 400 MHx Front side bus u Intel® E7500 Chipset u 6 PCI-X slots u 4 independent PCI buses u Can select:  64 bit 66 MHz PCI  100 MHz PCI-X  133 MHz PCI-X u Mbit Ethernet u Adaptec AIC-7899W dual channel SCSI u UDMA/100 bus master/EIDE channels  data transfer rates of 100 MB/sec burst u Collaboration: Boston Ltd. (Watford) – SuperMicro Motherboards, CPUs, Intel GE NICs Brunel University – Peter Van Santen University of Manchester – Richard Hughes-Jones

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 10 Latency & Throughput: Intel Pro/1000 on P4DP6  Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas)  CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz  RedHat 7.2 Kernel Latency high but smooth Indicates Interrupt coalescence Slope us/byte, Expect: PCI GigE0.008 PCI us/byte Max throughput 950Mbit/s Some throughput drop for packets >1000 bytes tests_Boston.ppt

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 11 New External Connectivity New External Connectivity  6 * 155 Mbit links  2.5Gbit line installed  IP commodity peer in London  Research traffic over 2.5G bit  Peer in Hudson St.  622 Mbit to Esnet  622 Mbit to Abilene.

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 12 Connectivity to Europe : Geant Connectivity to Europe : Geant  Start mid November 2001  UKERNA switched off TEN Dec 2001

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 13 ICFAMON Plot from DL to CERN for 18 th Feb to 3 rd Mar 2002 Connectivity to Europe Connectivity to Europe  UK Dante Access link 2.5 Gbit POS  Remember 19 th Oct to 1 st Nov Mbit Access link over loaded Sustained rate 130 Mbit

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 14 Monitoring: US Traffic Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US 7 day periods 1 hour averages Weekend-Before Weekday-After Weekday-Before 14 Jan 2002 (800Mbit/s) peak 86% of total 930 Mbit 17 Jan 2002 Peering altered 22 Jan 22 Jan 2002 Weed day peak 175 Mbit/s

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 15 Monitoring: US Traffic Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US 7 Dec 2001 (900kbit/s) 29 Jan 2002 (175kbit/s) peak is 88% of total BW 930 Mbit 10 minute averages 7 days 1 hour averages

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 16 DataGrid Network Monitoring u Several tools in use – plugged into a coherent structure:  PingER, RIPE one way times, iperf, UDPmon, rTPL, GridFTP, and NWS prediction engine u continuous tests for last few months to selected sites:  DL Man RL UCL CERN Lyon Bologna SARA NBI SLAC … uThe aims of monitoring for the Grid:  to inform Grid applications, via the middleware, of the current status of the network – input for resource broker and scheduling  to identify fault conditions in the operation of the Grid  to understand the instantaneous, day-to-day, and month-by-month behaviour of the network – provide advice on configuration etc. uNetwork information published in LDAP schema – new self-defining uCost Function in development – collaboration with WP1 & WP2 uWill be used by UK GridPP and e-science sites and non HEP WPs uLinks to the US

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 17 Local Network Monitoring Store & Analysis of Data (Access) Access to current and historic data and metrics via the Web, i.e. WP7 NM Pages, access to metric forecasts Backend LDAP script to fetch metrics Monitor process to push metrics local LDAP Server Grid Application access via LDAP Schema to - monitoring metrics; - location of monitoring data. PingER (RIPE TTB) iperf rTPL NWS etc LDAP Schema Grid Apps GridFTP Network Monitoring Architecture Robin Tasker

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 18 Network Monitoring Components PingNetmonUDPmoniPerfRipe Cron script plot Table LDAP raw control Cron script control Cron script plot Table LDAP rawplot Table LDAP raw WEB DisplayAnalysisGrid BrokerPredictions Web I/f Scheduler Tool Clients LDAP raw LDAP raw

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 19 Network Monitoring

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 20 Network Monitoring: Ping

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 21 Network Monitoring: Iperf (TCP)

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 22 HI Performance UDP Man - RAL Gigabit I/f Latency 5.2 mS Slope us/byte or 2 PCs expect: PCI GigE0.008 PCI Total us/byte 7 routers extra links 3 GigE G PoS Mbit0.012 Total Structure seen: Period 88 bytes Variation 150 – 190 us Max throughput 550Mbit/s Some throughput drop for packets < 20 us spacing  Manc 64bit 66 MHz PCI RedHat 7.1 Kernel NIC: NetGear  RAL RedHat 7.1 Kernel NIC: Intel pro 1000

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 23 HI Performance UDP Man - UvA Gigabit I/f Latency mS Slope us/byte For 2 PCs expect: PCI GigE0.008 PCI Total us/byte n routers extra links ?? No Structure seen: Throughput 825Mbit/s 1400 bytes Some throughput drop for packets < 20 us spacing Throughput 725Mbit/s 1200 bytes  Manc Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset  CPU: PIII 800 MHz PCI:64 bit 66 MHz  RedHat 7.1 Kernel  NIC: NetGear  UvA RedHat 7.1 Kernel ?  NIC: NetGear?

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 24 iperf TCP & UDP throughput MAN-SARA From 20 Oct 01 Iperf TCP throughput Mbit/s ucl – sara byte buffer Forecast UDPmon throughput Mbit/s man – sara 300 * 1400 byte frames

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 25 iperf & Pinger UK-Bologna From 20 Oct 01 Iperf throughput ucl – Bologna byte buffer Forecast in green PingER rtt (ms) dl – Bologna 1000 byte packet Forecast

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 26 Geant Enabled Routing Stable Iperf Throughput Mbit/s UCL – SARA byte buffer UDPmon Loss Throughput Mbit/s MAN – SARA iperf throughput UCL-SARA From 1 Nov 01 – Geant Operational

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 27 Iperf mem-mem vs file copy disk to disk Les Cottrell SLAC Iperf TCP Mbits/s File copy disk-to-disk Fast Ethernet OC3 Disk limited Over 60Mbits/s iperf >> file copy

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 28 Don’t Forget Involvement with: u GGF u US Grids PPDG, iVDGL uUKQCD UKDMC (dark matter) MINOS uAstroGRID uAccessGRID u E-science Centres u Optical “Lambda Switching” Projects u Collaborations with UKERNA, Dante, Terena …

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 29 More Information Some URLs  PPNCG Home page with Stop Press:  and  DataGrid WP7 Networking:  DataGrid WP7 EDG Monitoring:  IEPM PingER home site:  IEPM-BW site: