Progress on EXPReS at JBO EXPReS Board Meeting, Utrecht, Jan 2008 Ralph Spencer and the JBO team

Progress with iBOBs at Jodrell Contents Outline of JBO project Manpower Protocol work Progress on iBOBs Handover to Richard. Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell Outline of Work at JBO NA4 Outreach work – Alastair Gunn - completed SA1 Connection of 4 e-MERLIN telescopes to JIVE (see under Fabric) SA2: The 10 Gbps data link equipment available, awaiting cross connection in Manchester between academic and commercial links Fabric: Protocols for high speed networks –Stephen Kershaw : PFLDNet 2008 Manchester 5-7 March http://www.hep.man.ac.uk/PFLDnet2008/aboutManchester.php Fabric: Input to e-MERLIN correlator at 4 Gbps Jan 2008 Progress with iBOBs at Jodrell

E-MERLIN---e-VLBI system Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell Manpower Simon and Richard have left the team at JBO (though both still involved with EXPReS) Current contract for Jonathan expires November 2008 0.5 FTE in SA1 available for PDRA to work on testing NA4 Outreach work completed, using 0.07 of 0.5 FTE only Station boards (VSI interfaces) expected summer 2008 Suggest transferring 0.43 FTE from NA4 to Fabric If we get a 6 month no cost extension we will be able to Keep Jonathan on the project as a PDRA until the end of July 2009 Allow sufficient time for testing the input and output to the e-MERLIN correlator in eVLBI runs Keep on Stephen for a month or so this summer to help run the tests. Jan 2008 Progress with iBOBs at Jodrell

January 2008 Network protocols Stephen Kershaw To add: Stacks graphs from PFLDNET (indiv & tcpdelay) Stephen Kershaw

Progress with iBOBs at Jodrell TCP Is TCP suitable for transferring real-time data? If data is lost TCP will slow transmission and retransmit. Demonstrate the feasibility of using TCP for real-time applications Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems) We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings. Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell CUBIC TCP variants TCP using different algorithms for the window/throughput reduction Some variants allow more efficient use of bandwidth and appear more suitable for real-time applications HighSpeed Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems) We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings. Jan 2008 Progress with iBOBs at Jodrell

TCP variants Orders of magnitude improvements for speed of recovery with many variants Orders of magnitude reduction in buffering Evident bursty behaviour – concerns for highly loaded packet switched networks Note spikes due to bursts of data Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems) We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings. Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell Multicast Multicast reduces load on networks when sending to multiple hosts Test of multicast topology Multicast UDP stream over JANET academic network 100 2.2Mbps multicast IPTV streams originating from London, UK End-hosts on Manchester University campus network All Cisco network hardware, 100Mbps interfaces Measurement of inter-packet arrival times Time from arrival of packet ‘n’ to arrival of packet ‘n+1’ Expected ~6ms for one 2.2Mbps stream Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell Multicast Latency spikes of up to 0.5s at intervals of 60s Unexpected behaviour may limit the application of this technology Correlation between streams and between PCs shows a common source of delay Note that there is a variation with time of day (left edge is 7pm) – assumption is that it varies slightly with network load Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell Multicast Test of multicast topology - results Multicast traffic routing is not as efficient as unicast on this network Potential concerns for other networks Does not look suitable for eVLBI Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell DCCP Stable kernel Require tuning of virtual memory settings to prevent kernel panics Experimental kernels Revisions of late November 2007 address some memory and performance issues – yet to be tested on our systems. CCID4 in development Performance CCID2 can attain 1GE line-rate but it is not a turn-key thing Future work Test 2.6.24 kernel and recent DCCP git tree – not mature enough yet for a definitive statement about suitability Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems) We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings. Jan 2008 Progress with iBOBs at Jodrell

Progress on Digital Interfaces iBOBs and all that Jonathan Hargreaves/Richard Hughes-Jones

Progress with iBOBs at Jodrell The iBOB System As a transmitter: Now able to transfer data at line rate to a test PC over a 10 GE (CX-4) link Needs A/D connection and interface (Onsala/Metsahovi) As a receiver Needs to interface to VSI chip on station board As a network test system See also Richard’s talk Fujitsu switch ordered Jan 2008 Progress with iBOBs at Jodrell

Progress with iBOBs at Jodrell JBO Receiver: Status Recent Progress Sixteen packet FIFO implemented in the iBOB’s FPGA Even though packets are transmitted at 4Gbps they might ‘bunch up’ and arrive at line speed. The FIFO is intended to smooth them out again The 10Gb transceiver can run at 200MHz – enough to handle line speed, but the SRAM bandwidth is 167MHz for both read and write. Read out at 4Gbps consumes 64MHz of that. More resource than needed Data flows from the iBOB to the Station Board defined (see next slide) Data readout from the SRAM synchronised to the correlator clock, when present When correlator clock is not present, iBOB defaults to test mode using the internal clock VSI chip will buffer and reclock data and route it to the Station Board input Next Tasks Test the new code in hardware using another iBOB to generate test packets Write firmware for the VSI chip and either simulate or test on remote hardware Jan 2008 Progress with iBOBs at Jodrell

JBO Receiver: Block Diagram 10GB RX ‘Yellow box’ on the Simulink design 4GB/s CX4 FIFO 16k deep Holds up to 16 packets in case short bursts arrive at line rate . SRAM Control Re-orders out of order packets Counts lost packets Reads data out to the correlator at a steady 4Gbps SRAM 512k x 64 bit (512 packets = 9ms at 4Gbps) ‘VSI chip’ Buffer and reclock data. Reroute it to the left and right polarisation station board (SB) inputs. Clock Control Lock 128MHz clock to correlator DATA 32 bit DATA 64 bit Correlator CLK ADDR DATA 2 x MDR80 ‘VSI’ cables Station Board 200MHz System Clock 128MHz DATA to SB Input iBOB Jan 2008 Progress with iBOBs at Jodrell

Network Testing Device: Status Recent Progress System clock increased from 100MHz to 200MHz to allow line speed data flows at 10Gbps (JH) Added jumper bank to select ethernet IP and MAC address (JH & RHJ) All registers, test modes and results can be accessed via ethernet control/response protocol (RHJ) PC based software provides user interface to the ethernet control link (RHJ) Ethernet based control supersedes the TinySH command line and is seen as a prototype for control of iBOBs sending real data Currently testing iBOB to iBOB transfers on bench Next Tasks Count lost packets, out of order packets and duplicate packets Update real time field correctly each second – currently it is fixed over a burst Test over network Jan 2008 Progress with iBOBs at Jodrell