Presentation is loading. Please wait.

Presentation is loading. Please wait.

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02)

Published byIsaac Francis Modified over 9 years ago

Similar presentations

Presentation on theme: "GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02)"— Presentation transcript:

1 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02) Collaboration: Boston Ltd. (Watford) – SuperMicro Motherboards, CPUs, Intel GE NICs Brunel University – Peter Van Santen University of Manchester – Richard Hughes-Jones www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt

2 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester The Measurements (1) uLatency uRound trip times measured using Request-Response UDP frames uLatency as a function of frame size Slope gives sum of individual data transfer rates end-to-end Mem copy + pci + Gig Ethernet + pci + mem copy uHistograms of individual measurements

3 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester The Measurements (2) uUDP Throughput uSend a burst of UDP frames spaced at regular intervals uVary the frame size and the frame transmit spacing uRecord The time to send and the time to receive the frames The number received, the number lost, number out of order The received inter-packet spacing CPU load, Number of interrupts Zero stats OK done ●●● Get remote statistics Send statistics Send data frames at regular intervals ●●● Time to send Time to receive Inter-packet time

4 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester The Measurements (3) uPCI Activity uLogic Analyzer with PCI Probe cards in sending PC Gigabit Ethernet Fiber Probe Card PCI Probe cards in receiving PC Gigabit Ethernet Probe CPU mem chipset NIC CPU mem NIC chipset Logic Analyser Display

5 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Alteon AceNIC Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14

6 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester UDP Throughput: Alteon AceNIC Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14

7 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Alteon AceNIC Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit RedHat 7.1 Kernel 2.4.14 ALT33102 PCI 33 MHz 1400 bytes sent Wait 16 us ALT66101 66 MHz 1400 bytes sent Wait 16 us NIC cannot sustain 66 MHz Send PCI Receive PCI

8 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 Latency low good Latency well behaved Slope 0.0252 us/byte Expect: PCI0.00758 GigE0.008 PCI0.00758 0.0236 us/byte

9 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester UDP Throughput: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 Max throughput 690Mbit/s No packet loss Packet loss during drop

10 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 SK300 1400 bytes sent Wait 100 us ~8 us for send or receive Gigabit Ethernet frame

11 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 SK301 1400 bytes sent Wait 20 us Sk303 1400 bytes sent Wait 10 us Frames are back-to-back Cannot go any faster ! Gig Eth frames back to back

12 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 Latency high Latency well behaved Slope 0.0187 us/byte Expect: PCI0.00188 GigE0.008 PCI0.00188 0.0118 us/byte

13 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 IT66M200 64 bytes sent CSR time: 1.75 us Data time 0.25 us Interrupt delay:~70 us 1400 response

14 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 Max throughput 910Mbit/s No packet loss Packet loss during drop

15 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 losses occur in groups ~50 pkts every 140

16 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 IT66M212 1400 bytes sent Wait 11 us ~4.7us on send PCI bus PCI bus ~45% occupancy ~ 3.25 us on PCI for data recv IT66M212 1400 bytes sent Wait 11 us Packets lost Action of pause packet?

17 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.2 Latency high Slope 0.0195 us/byte Expect: PCI0.00188 GigE0.008 PCI0.00188 0.0118 us/byte

18 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.2 Max throughput 950Mbit/s No packet loss Negligible Packet loss

19 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.2 IT66M220 1400 bytes sent Wait 1000 us CSR time: 12.25 us Data time 5.0 us Interrupt delay:~79 us IT66M224 1400 bytes sent Wait 100 us Detail Chipset limits PCI transfers with STOPs Try i870 Chipset

20 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4CD6+ Motherboard: SuperMicro P4CD6+ Chipset: Intel i860 CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.2 IT66M221 1400 bytes sent Wait 11 us

21 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 on IBM board Motherboard: IBM das Chipset:: ServerWorks CNB20LE CPU: Dual PIII 1GHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 Latency high Latency well behaved Slope 0.0206 us/byte Expect: PCI0.00376 GigE0.008 PCI0.00376 0.0155 us/byte

22 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 on IBM board Motherboard: IBM das Chipset:: ServerWorks CNB20LE CPU: Dual PIII 1GHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 Max throughput 930Mbit/s No packet loss Packet loss during drop

23 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on IBM board Motherboard: IBM das Chipset:: ServerWorks CNB20LE CPU: Dual PIII 1GHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 uva64m02 1400 bytes sent Wait 11 us ~9.3us on send PCI bus PCI bus ~82% occupancy ~ 5.9 us on PCI for data recv.

24 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Latency: Intel Pro/1000 on P4DP6 Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14 Latency high but smooth Indicates Interrupt coalescence Slope 0.0136 us/byte Expect: PCI0.00188 GigE0.008 PCI0.00188 0.0118 us/byte

25 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Throughput: Intel Pro/1000 on P4DP6 Max throughput 950Mbit/s Some throughput drop for packets >1000 bytes Packet loss small 800 – 1000 byte packets Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

26 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4DP6 ITP4007 1400 bytes sent Wait 1000 us Send: CSR time: 2.0 us Send: Data time 3.25 us Recv: Data time 2.2 us Slot 3 to slot 5 ITP4001 Detail of 1400 bytes sent CSR time 2.2 us Data time 3.2 us Slot 4 to slot 4 Small differences between slots Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slots: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

27 GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4DP6 ITP4010 1400 bytes sent Wait 8 us ~5.14us on send PCI bus PCI bus ~68% occupancy ~ 2 us on PCI for data recv Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 3-5: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

Download ppt "GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02)"

Similar presentations

MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.

MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.
DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)

DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)
CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/

CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester
ER2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets 256-500 Mbit/s on the production network.

ER2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets Mbit/s on the production network.
JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.

JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.
GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.
20th-21st June 2005 NeSC Edinburgh End-user systems: NICs, MotherBoards, Disks, TCP Stacks & Applications Richard Hughes-Jones.

20th-21st June 2005 NeSC Edinburgh End-user systems: NICs, MotherBoards, Disks, TCP Stacks & Applications Richard Hughes-Jones.
ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.

ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.
Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.

Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.
Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.

Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.
High-Performance Throughput Tuning/Measurements Davide Salomoni & Steffen Luitz Presented at the PPDG Collaboration Meeting, Argonne National Lab, July.

High-Performance Throughput Tuning/Measurements Davide Salomoni & Steffen Luitz Presented at the PPDG Collaboration Meeting, Argonne National Lab, July.
CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.

CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.
IEEE Real Time 2007, Fermilab, 29 April – 4 May R. Hughes-Jones Manchester 1 Using FPGAs to Generate Gigabit Ethernet Data Transfers & The Network Performance.

IEEE Real Time 2007, Fermilab, 29 April – 4 May R. Hughes-Jones Manchester 1 Using FPGAs to Generate Gigabit Ethernet Data Transfers & The Network Performance.
DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)

DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)
JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.
T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then.

T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester then.
CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones.

CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones.
DataTAG Meeting CERN 7-8 May 03 R. Hughes-Jones Manchester 1 High Throughput: Progress and Current Results Lots of people helped: MB-NG team at UCL MB-NG.

DataTAG Meeting CERN 7-8 May 03 R. Hughes-Jones Manchester 1 High Throughput: Progress and Current Results Lots of people helped: MB-NG team at UCL MB-NG.
PFLDNet Argonne Feb 2004 R. Hughes-Jones Manchester 1 UDP Performance and PCI-X Activity of the Intel 10 Gigabit Ethernet Adapter on: HP rx2600 Dual Itanium.

PFLDNet Argonne Feb 2004 R. Hughes-Jones Manchester 1 UDP Performance and PCI-X Activity of the Intel 10 Gigabit Ethernet Adapter on: HP rx2600 Dual Itanium.
© 2006 Open Grid Forum Interactions Between Networks, Protocols & Applications HPCN-RG Richard Hughes-Jones OGF20, Manchester, May 2007,

© 2006 Open Grid Forum Interactions Between Networks, Protocols & Applications HPCN-RG Richard Hughes-Jones OGF20, Manchester, May 2007,

Similar presentations

Ads by Google