GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02)

Slides:

Advertisements

Similar presentations

MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.

Advertisements

DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)

CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester

ER2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets Mbit/s on the production network.

JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.

20th-21st June 2005 NeSC Edinburgh End-user systems: NICs, MotherBoards, Disks, TCP Stacks & Applications Richard Hughes-Jones.

ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.

Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.

Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.

High-Performance Throughput Tuning/Measurements Davide Salomoni & Steffen Luitz Presented at the PPDG Collaboration Meeting, Argonne National Lab, July.

CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.

IEEE Real Time 2007, Fermilab, 29 April – 4 May R. Hughes-Jones Manchester 1 Using FPGAs to Generate Gigabit Ethernet Data Transfers & The Network Performance.

DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.

T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester then.

CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones.

DataTAG Meeting CERN 7-8 May 03 R. Hughes-Jones Manchester 1 High Throughput: Progress and Current Results Lots of people helped: MB-NG team at UCL MB-NG.

PFLDNet Argonne Feb 2004 R. Hughes-Jones Manchester 1 UDP Performance and PCI-X Activity of the Intel 10 Gigabit Ethernet Adapter on: HP rx2600 Dual Itanium.

© 2006 Open Grid Forum Interactions Between Networks, Protocols & Applications HPCN-RG Richard Hughes-Jones OGF20, Manchester, May 2007,

Slide: 1 Richard Hughes-Jones CHEP2004 Interlaken Sep 04 R. Hughes-Jones Manchester 1 Bringing High-Performance Networking to HEP users Richard Hughes-Jones.

CdL was here DataTAG CERN Sep 2002 R. Hughes-Jones Manchester 1 European Topology: NRNs & Geant SuperJANET4 CERN UvA Manc SURFnet RAL.

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 1 VLBI & Protocols vlbi_udp Multiple Flow Tests Richard Hughes-Jones The University.

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 1 Networking in Under 30 Minutes ! Richard Hughes-Jones, University of Manchester.

July 2000 PPNCG Meeting R. Hughes-Jones Performance Measurements of LANs MANs and SuperJANET III This is PRELIMINARY uBaseline data for Grid development.

Can Google Route? Building a High-Speed Switch from Commodity Hardware Guido Appenzeller, Matthew Holliman Q2/2002.

02 nd April 03Networkshop Managed Bandwidth Next Generation F. Saka UCL NETSYS (NETwork SYStems centre of excellence)

13th-14th July 2004 University College London End-user systems: NICs, MotherBoards, TCP Stacks & Applications Richard Hughes-Jones.

Protocol-Dependent Message-Passing Performance on Linux Clusters Dave Turner – Xuehua Chen – Adam Oline This work is funded by the DOE MICS office.

Sven Ubik, Petr Žejdl CESNET TNC2008, Brugges, 19 May 2008 Passive monitoring of 10 Gb/s lines with PC hardware.

Slide: 1 Richard Hughes-Jones Summer School, Brasov, Romania, July 2005, R. Hughes-Jones Manchester 1 TCP/IP and Other Transports for High Bandwidth Applications.

CSC 311 IEEE STANDARD ETHERNET Common Bus topology Uses CSMA/CD Named after “ether”, the imaginary substance many once believed occupied all of space.

15-Feb-02PvS Brunel Report, GridPP 3 Cambridge 1 Brunel University ECE Brunel Grid Activities Report Peter van Santen Distributed and Grid Computing Group.

A TCP/IP transport layer for the DAQ of the CMS Experiment Miklos Kozlovszky for the CMS TriDAS collaboration CERN European Organization for Nuclear Research.

EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 1 FABRIC 4 Gigabit Work & VLBI-UDP Performance and Stability. Richard Hughes-Jones The.

“ PC  PC Latency measurements” G.Lamanna, R.Fantechi & J.Kroon (CERN) TDAQ WG –

Slide: 1 Richard Hughes-Jones e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester 1 TCP/IP Overview & Performance Richard Hughes-Jones The University.

Linux Servers with JASMine K. Edwards, A. Kowalski, S. Philpott HEPiX May 21, 2003.

Slide: 1 Richard Hughes-Jones Mini-Symposium on Optical Data Networking, August 2005, R. Hughes-Jones Manchester 1 Using TCP/IP on High Bandwidth Long.

High TCP performance over wide area networks Arlington, VA May 8, 2002 Sylvain Ravot CalTech HENP Working Group.

Platform Architecture Lab USB Performance Analysis of Bulk Traffic Brian Leete

1 Network Performance Optimisation and Load Balancing Wulf Thannhaeuser.

ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 1 VLBI Data Transfer Tests Recent and Current Work. Richard Hughes-Jones The University.

MB - NG MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester MB – NG SuperJANET4 Development Network SuperJANET4 Production Network Leeds RAL / UKERNA RAL.

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 1 Protocols DCCP and dccpmon. Richard Hughes-Jones The University of Manchester.

Robin HJ & R. Hughes-Jones Manchester Sep 1999 Gigabit Ethernet in Ptolemy Status Sep 99 : Stars that exist : –GigEChipTranslate between GigEPacket and.

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard.

PFLDNet Workshop February 2003 R. Hughes-Jones Manchester Some Performance Measurements Gigabit Ethernet NICs & Server Quality Motherboards Richard Hughes-Jones.

Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 1 Collaborations in Networking and Protocols HEP and Radio Astronomy Richard Hughes-Jones.

High bit rate tests between Manchester and JIVE Looking at data rates attainable with UDP along with packet loss and reordering statistics Simon Casey,

DataGrid WP7 Meeting Amsterdam Nov 01 R. Hughes-Jones Manchester 1 UDPmon Measuring Throughput with UDP  Send a burst of UDP frames spaced at regular.

LNL 1 SADIRC2000 Resoconto 2000 e Richieste LNL per il 2001 L. Berti 30% M. Biasotto 100% M. Gulmini 50% G. Maron 50% N. Toniolo 30% Le percentuali sono.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.

Networks ∙ Services ∙ People Richard-Hughes Jones eduPERT Training Session, Porto A Hands-On Session udpmon for Network Troubleshooting 18/06/2015.

Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.

1 eVLBI Developments at Jodrell Bank Observatory Ralph Spencer, Richard Hughes- Jones, Simon Casey, Paul Burgess, The University of Manchester.

L1/HLT trigger farm Bologna setup 0 By Gianluca Peco INFN Bologna Genève,

16 th IEEE NPSS Real Time Conference 2009 IHEP, Beijing, China, 12 th May, 2009 High Rate Packets Transmission on 10 Gbit/s Ethernet LAN Using Commodity.

Recent experience with PCI-X 2.0 and PCI-E network interfaces and emerging server systems Yang Xia Caltech US LHC Network Working Group October 23, 2006.

DiSCoV Fall 2008 Paul A. Farrell Cluster Computing 1 Improving Cluster Performance Performance Evaluation of Networks.

Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI Richard Hughes-Jones.

DataGrid WP7 Meeting Jan 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress)

MB MPLS MPLS Technical Meeting Sep 2001 R. Hughes-Jones Manchester SuperJANET Development Network Testbed – Cisco GSR SuperJANET4 C-PoP – Cisco GSR.

CALICE TDAQ Application Network Protocols 10 Gigabit Lab

R. Hughes-Jones Manchester

Data Transfer Node Performance GÉANT & AENEAS

Evolution of S-LINK to PCI interfaces

Mar 2001 ATLAS T2UK Meeting R. Hughes-Jones

MB-NG Review High Performance Network Demonstration 21 April 2004

MB – NG SuperJANET4 Development Network

Presentation transcript:

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02) Collaboration: Boston Ltd. (Watford) – SuperMicro Motherboards, CPUs, Intel GE NICs Brunel University – Peter Van Santen University of Manchester – Richard Hughes-Jones

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester The Measurements (1) uLatency uRound trip times measured using Request-Response UDP frames uLatency as a function of frame size Slope gives sum of individual data transfer rates end-to-end Mem copy + pci + Gig Ethernet + pci + mem copy uHistograms of individual measurements

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester The Measurements (2) uUDP Throughput uSend a burst of UDP frames spaced at regular intervals uVary the frame size and the frame transmit spacing uRecord The time to send and the time to receive the frames The number received, the number lost, number out of order The received inter-packet spacing CPU load, Number of interrupts Zero stats OK done ●●● Get remote statistics Send statistics Send data frames at regular intervals ●●● Time to send Time to receive Inter-packet time

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester The Measurements (3) uPCI Activity uLogic Analyzer with PCI Probe cards in sending PC Gigabit Ethernet Fiber Probe Card PCI Probe cards in receiving PC Gigabit Ethernet Probe CPU mem chipset NIC CPU mem NIC chipset Logic Analyser Display

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Alteon AceNIC Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester UDP Throughput: Alteon AceNIC Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Alteon AceNIC Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit RedHat 7.1 Kernel ALT33102 PCI 33 MHz 1400 bytes sent Wait 16 us ALT MHz 1400 bytes sent Wait 16 us NIC cannot sustain 66 MHz Send PCI Receive PCI

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel Latency low good Latency well behaved Slope us/byte Expect: PCI GigE0.008 PCI us/byte

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester UDP Throughput: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel Max throughput 690Mbit/s No packet loss Packet loss during drop

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel SK bytes sent Wait 100 us ~8 us for send or receive Gigabit Ethernet frame

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel SK bytes sent Wait 20 us Sk bytes sent Wait 10 us Frames are back-to-back Cannot go any faster ! Gig Eth frames back to back

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel Latency high Latency well behaved Slope us/byte Expect: PCI GigE0.008 PCI us/byte

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel IT66M bytes sent CSR time: 1.75 us Data time 0.25 us Interrupt delay:~70 us 1400 response

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel Max throughput 910Mbit/s No packet loss Packet loss during drop

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel losses occur in groups ~50 pkts every 140

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel IT66M bytes sent Wait 11 us ~4.7us on send PCI bus PCI bus ~45% occupancy ~ 3.25 us on PCI for data recv IT66M bytes sent Wait 11 us Packets lost Action of pause packet?

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel Latency high Slope us/byte Expect: PCI GigE0.008 PCI us/byte

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel Max throughput 950Mbit/s No packet loss Negligible Packet loss

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel IT66M bytes sent Wait 1000 us CSR time: us Data time 5.0 us Interrupt delay:~79 us IT66M bytes sent Wait 100 us Detail Chipset limits PCI transfers with STOPs Try i870 Chipset

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel IT66M bytes sent Wait 11 us

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 on IBM board Motherboard: IBM das Chipset:: ServerWorks CNB20LE CPU: Dual PIII 1GHz PCI:64 bit 33 MHz RedHat 7.1 Kernel Latency high Latency well behaved Slope us/byte Expect: PCI GigE0.008 PCI us/byte

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 on IBM board Motherboard: IBM das Chipset:: ServerWorks CNB20LE CPU: Dual PIII 1GHz PCI:64 bit 33 MHz RedHat 7.1 Kernel Max throughput 930Mbit/s No packet loss Packet loss during drop

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on IBM board Motherboard: IBM das Chipset:: ServerWorks CNB20LE CPU: Dual PIII 1GHz PCI:64 bit 33 MHz RedHat 7.1 Kernel uva64m bytes sent Wait 11 us ~9.3us on send PCI bus PCI bus ~82% occupancy ~ 5.9 us on PCI for data recv.

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 on P4DP6 Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel Latency high but smooth Indicates Interrupt coalescence Slope us/byte Expect: PCI GigE0.008 PCI us/byte

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 on P4DP6 Max throughput 950Mbit/s Some throughput drop for packets >1000 bytes Packet loss small 800 – 1000 byte packets Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4DP6 ITP bytes sent Wait 1000 us Send: CSR time: 2.0 us Send: Data time 3.25 us Recv: Data time 2.2 us Slot 3 to slot 5 ITP4001 Detail of 1400 bytes sent CSR time 2.2 us Data time 3.2 us Slot 4 to slot 4 Small differences between slots Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slots: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4DP6 ITP bytes sent Wait 8 us ~5.14us on send PCI bus PCI bus ~68% occupancy ~ 2 us on PCI for data recv Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 3-5: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel