JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University of Manchester DataGrid WP7 – Dante Tests on the GÉANT Core End-2-End Measurements from the 4 th Year VLBI Project at Manchester DiskPack-2-Memory Throughput and PCI Activity in a Mark5 PC Update on 1 and 10 Gigabit Ethernet NICs in the PC

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester DataGrid WP7 – Dante Tests on the GÉANT Core Set-up Supermicro PC in: London GEANT PoP Amsterdam GEANT PoP Smartbits in: London GEANT PoP Frankfurt GEANT PoP Long link UK-SE-DE2-IT-CH-FR-BE-NL Short Link UK-FR-BE-NL

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Tests GÉANT Core: UDP throughput UDP Throughput London-Amsterdam Available BW to packet on wire Then 1/t Wire rate 998 Mbit/s for packets > 1400 bytes Packet Loss None for large packets Dips in BW lined to packet loss SysKonnect NIC int. per packet CPU load important

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Tests GÉANT Core: Packet re-ordering Effect of Packet size London-Amsterdam Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1% Re-order Distribution

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Tests GÉANT Core: Packet re-ordering Effect of LBE background Amsterdam-London BE Test flow Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1% Re-order Distributions:

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester VLBI Project: Test Topology SuperJANET4 Jodrell Manchester SURFnet JIVE Dwingaloo Adam Mathews Steve O’Toole Univ of Manchester

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Manchester to Dwingeloo 2.0G Hz Xeon 1.2 GHz PIII Re-ordering vs Offered Load VLBI Project: Throughput

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester 1472 byte Packets man -> JIVE FWHM 22 µs (B2B 3 µs ) VLBI Project: Jitter & 1-way Delay 1-way Delay – note the packet loss (points with 0 –way delay)

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Measure the time between lost packets in the time series of packets sent. Lost 1410 in 0.6s Is it a Poisson process? Assume Poisson is stationary λ(t) = λ Use Prob. Density Function: P(t) = λ e -λt Mean λ = 2360 / s [426 µs] Plot log: slope expect Could be additional process involved VLBI Project: Packet Loss Distribution

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Divide time series of packets into 1000 slices of 50 packets Total lost packets 1410 Average number / slice = 1.4 Calc Poisson Probability P(n, µ) = µ n e -µ n! Curves close but not exact Could be more than 1 process VLBI Project: Packet Loss – Is it Poisson?

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Aggregated Variance Method Divide time series length N into blocks of size m Calc mean of each section Xm(k) k= 1 … N/m Calc variance VXm of these Xm(k) Vary m size of the blocks Plot on log-log & fit slope β Hurst parameter H β = 2H -2 Measure: β = which gives H H =1 no long range dependence VLBI Project: Packet Loss – Long Range Effects?

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Traffic Flows uManchester – NetNorthWest - SuperJANET Access links Two 1 Gbit/s uAccess links: SJ4 to GÉANT GÉANT to SurfNet

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Throughput & PCI transactions on the Mark5 PC: Read / Write n bytes Wait time time  uMark5 uses Supermicro P3TDLE 1.2 GHz PIII Mem bus 133/100 MHz 2 *64bit 66 MHz PCI 4 32bit 33 MHz PCI SuperStor NIC Input Card IDE Disc Pack Ethernet Logic Analyser Display

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester PCI Activity: Read 1 data block 0 wait time uData block 131,072 bytes uRead consists of: Setup CSRs Data movement Update CSRs uData block contains PCI bursts 4096 bytes long uFor 0 wait between reads: Data block 608µs long Then 655µs gap uPCI transfer rate 830 Mbit/s uRead_sstor rate 778 Mbit/s (97 Mbyte/s) CSR Access PCI Burst Data Block

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester PCI Activity: Read Multiple data blocks 0 wait uRead bytes uEach Data block: Setup CSRs Data movement Update CSRs uFor 0 wait between reads: Data blocks ~600µs long take ~6 ms Then 744µs gap uPCI transfer rate 1188Mbit/s (148.5 Mbytes/s) uRead_sstor rate 778 Mbit/s (97 Mbyte/s) uPCI bus occupancy: 68.44% uConcern about Ethernet Traffic 64 bit 33 MHz PCI needs ~ 82% for 930 Mbit/s Expect ~360 Mbit/s

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester PCI Activity: Read Throughput uFlat then 1/t dependance u~ 860 Mbit/s for Read blocks >= bytes uCPU load ~20% uConcern about CPU load needed to drive Gigabit link

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester 10 GigEthernet: Throughput u1500 byte MTU gives ~ 2 Gbit/s uUsed byte MTU max user length uDataTAG Supermicro PCs uDual 2.2 GHz Xenon CPU FSB 400 MHz uPCI-X mmrbc 512 bytes uwire rate throughput of 2.9 Gbit/s uCERN OpenLab HP Itanium PCs uDual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz uPCI-X mmrbc 512 bytes uwire rate of 5.7 Gbit/s uSLAC Dell PCs giving a uDual 3.0 GHz Xenon CPU FSB 533 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.4 Gbit/s

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester 10 GigEthernet: PCI-X Activity uSupermicro P4DP8-2G motherboard Packet transmission Memory to NIC Packet reception NIC to Memory CSR Access Interrupt PCI-X segment 512 bytes Transfer of bytes

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester 10 GigEthernet: Tuning PCI-X

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester 10 GigEthernet at SC2003 BW Challenge (Phoenix) uThree Server systems with 10 GigEthernet NICs uUsed the DataTAG altAIMD stack 9000 byte MTU uStreams From SLAC/FNAL booth in Phoenix to: Pal Alto PAIX Chicago Starlight Amsterdam SARA

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Traffic QoS Classes on GÉANT Backbone uNormal Traffic uNormal Traffic + uLess Than Best Effort 2.0 Gbit/s uNormal Traffic + uRadio Astronomy Data 500 Mbit/s uNormal Traffic + uRadio Astronomy Data + uLess Than Best Effort 2.0 Gbit/s Max Throughput on 2.5 G PoS

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Some Measurements made during ER2002