1. Progress on EXPReS at JBO
EXPReS Board Meeting, Utrecht, Jan 2008
Ralph Spencer and the JBO team

2. Progress with iBOBs at Jodrell
Contents
Outline of JBO project
Manpower
Protocol work
Progress on iBOBs
Handover to Richard.
Jan 2008
Progress with iBOBs at Jodrell

3. Progress with iBOBs at Jodrell
Outline of Work at JBO
NA4 Outreach work – Alastair Gunn - completed
SA1 Connection of 4 e-MERLIN telescopes to JIVE (see under Fabric)
SA2: The 10 Gbps data link equipment available, awaiting cross connection in Manchester between academic and commercial links
Fabric: Protocols for high speed networks –Stephen Kershaw : PFLDNet 2008 Manchester 5-7 March
Fabric: Input to e-MERLIN correlator at 4 Gbps
Jan 2008
Progress with iBOBs at Jodrell

4. E-MERLIN---e-VLBI system
Jan 2008
Progress with iBOBs at Jodrell

5. Progress with iBOBs at Jodrell
Manpower
Simon and Richard have left the team at JBO (though both still involved with EXPReS)
Current contract for Jonathan expires November 2008
0.5 FTE in SA1 available for PDRA to work on testing
NA4 Outreach work completed, using 0.07 of 0.5 FTE only
Station boards (VSI interfaces) expected summer 2008
Suggest transferring 0.43 FTE from NA4 to Fabric
If we get a 6 month no cost extension we will be able to
Keep Jonathan on the project as a PDRA until the end of July 2009
Allow sufficient time for testing the input and output to the e-MERLIN correlator in eVLBI runs
Keep on Stephen for a month or so this summer to help run the tests.
Jan 2008
Progress with iBOBs at Jodrell

6. January 2008 Network protocols Stephen Kershaw To add:
Stacks graphs from PFLDNET (indiv & tcpdelay)
Stephen Kershaw

7. Progress with iBOBs at Jodrell
TCP
Is TCP suitable for transferring real-time data?
If data is lost TCP will slow transmission and retransmit.
Demonstrate the feasibility of using TCP for real-time applications
Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems)
We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings.
Jan 2008
Progress with iBOBs at Jodrell

8. Progress with iBOBs at Jodrell
CUBIC
TCP variants
TCP using different algorithms for the window/throughput reduction
Some variants allow more efficient use of bandwidth and appear more suitable for real-time applications
HighSpeed
Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems)
We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings.
Jan 2008
Progress with iBOBs at Jodrell

9. TCP variants
Orders of magnitude improvements for speed of recovery with many variants
Orders of magnitude reduction in buffering
Evident bursty behaviour – concerns for highly loaded packet switched networks
Note spikes due to bursts of data
Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems)
We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings.
Jan 2008
Progress with iBOBs at Jodrell

10. Progress with iBOBs at Jodrell
Multicast
Multicast reduces load on networks when sending to multiple hosts
Test of multicast topology
Multicast UDP stream over JANET academic network
Mbps multicast IPTV streams originating from London, UK
End-hosts on Manchester University campus network
All Cisco network hardware, 100Mbps interfaces
Measurement of inter-packet arrival times
Time from arrival of packet ‘n’ to arrival of packet ‘n+1’
Expected ~6ms for one 2.2Mbps stream
Jan 2008
Progress with iBOBs at Jodrell

11. Progress with iBOBs at Jodrell
Multicast
Latency spikes of up to 0.5s at intervals of 60s
Unexpected behaviour may limit the application of this technology
Correlation between streams and between PCs shows a common source of delay
Note that there is a variation with time of day (left edge is 7pm) – assumption is that it varies slightly with network load
Jan 2008
Progress with iBOBs at Jodrell

12. Progress with iBOBs at Jodrell
Multicast
Test of multicast topology - results
Multicast traffic routing is not as efficient as unicast on this network
Potential concerns for other networks
Does not look suitable for eVLBI
Jan 2008
Progress with iBOBs at Jodrell

13. Progress with iBOBs at Jodrell
DCCP
Stable kernel
Require tuning of virtual memory settings to prevent kernel panics
Experimental kernels
Revisions of late November 2007 address some memory and performance issues – yet to be tested on our systems.
CCID4 in development
Performance
CCID2 can attain 1GE line-rate but it is not a turn-key thing
Future work
Test kernel and recent DCCP git tree – not mature enough yet for a definitive statement about suitability
Development (not us) continues but modifications are slow to reach stable kernel (at least 2 point releases it seems)
We can achieve good throughput – ADD TRANSATLANTIC TPUT GRAPH - if we try hard to install new releases and tune VM settings.
Jan 2008
Progress with iBOBs at Jodrell

14. Progress on Digital Interfaces
iBOBs and all that
Jonathan Hargreaves/Richard Hughes-Jones

15. Progress with iBOBs at Jodrell
The iBOB System
As a transmitter:
Now able to transfer data at line rate to a test PC over a 10 GE (CX-4) link
Needs A/D connection and interface (Onsala/Metsahovi)
As a receiver
Needs to interface to VSI chip on station board
As a network test system
See also Richard’s talk
Fujitsu switch ordered
Jan 2008
Progress with iBOBs at Jodrell

16. Progress with iBOBs at Jodrell
JBO Receiver: Status
Recent Progress
Sixteen packet FIFO implemented in the iBOB’s FPGA
Even though packets are transmitted at 4Gbps they might ‘bunch up’ and arrive at line speed. The FIFO is intended to smooth them out again
The 10Gb transceiver can run at 200MHz – enough to handle line speed, but the SRAM bandwidth is 167MHz for both read and write. Read out at 4Gbps consumes 64MHz of that. More resource than needed
Data flows from the iBOB to the Station Board defined (see next slide)
Data readout from the SRAM synchronised to the correlator clock, when present
When correlator clock is not present, iBOB defaults to test mode using the internal clock
VSI chip will buffer and reclock data and route it to the Station Board input
Next Tasks
Test the new code in hardware using another iBOB to generate test packets
Write firmware for the VSI chip and either simulate or test on remote hardware
Jan 2008
Progress with iBOBs at Jodrell

17. JBO Receiver: Block Diagram
10GB RX
‘Yellow box’ on the
Simulink design
4GB/s CX4
FIFO
16k deep
Holds up to 16 packets in case short bursts arrive at line rate .
SRAM Control
Re-orders out of order packets
Counts lost packets
Reads data out to the correlator at a steady 4Gbps
SRAM
512k x 64 bit
(512 packets = 9ms at 4Gbps)
‘VSI chip’
Buffer and reclock data.
Reroute it to the left and right polarisation station board (SB) inputs.
Clock Control
Lock 128MHz clock to correlator
DATA 32 bit
DATA 64 bit
Correlator CLK
ADDR
DATA
2 x MDR80 ‘VSI’ cables
Station Board
200MHz System Clock
128MHz
DATA to SB Input
iBOB
Jan 2008
Progress with iBOBs at Jodrell

18. Network Testing Device: Status
Recent Progress
System clock increased from 100MHz to 200MHz to allow line speed data flows at 10Gbps (JH)
Added jumper bank to select ethernet IP and MAC address (JH & RHJ)
All registers, test modes and results can be accessed via ethernet control/response protocol (RHJ)
PC based software provides user interface to the ethernet control link (RHJ)
Ethernet based control supersedes the TinySH command line and is seen as a prototype for control of iBOBs sending real data
Currently testing iBOB to iBOB transfers on bench
Next Tasks
Count lost packets, out of order packets and duplicate packets
Update real time field correctly each second – currently it is fixed over a burst
Test over network
Jan 2008
Progress with iBOBs at Jodrell