1. SA1: Production e-VLBI service Arpad Szomoru JIVE

2. 2008 May 30 EXPReS Annual Review: SA1Slide #2 Outline e-VLBI science/test runs Operational improvements Global e-VLBI: the demonstrations Hard- and software developments

3. 2008 May 30 EXPReS Annual Review: SA1Slide #3 e-VLBI science/test runs 1 Target of Opportunity project (target: nearby supernova) 1 response to a triggered proposal (target: accreting X-ray binary in a special X-ray state) 2 exploratory projects (both targets were candidate radio counterparts to so far unidentified gamma-ray source) 1 Spectral line project (targets: candidate MASER sources) October deadline only had one triggered proposal Lack of triggers meant no science runs until March 2009...but, available time was used very well Different proposal/selection mechanism clearly needed 3rd year already much more productive

4. 2008 May 30 EXPReS Annual Review: SA1Slide #4 e-VLBI: operational improvements Overall improvement in second year: Huge increase in robustness, reliability and user-friendliness Not limited to e-operations Increase of operational data rate from 256 to 512 Mbps First experiments at ~1 Gbps (with European telescopes) Demonstrations of global e-VLBI with Chinese and Australian telescopes Solution of long-haul problems Inclusion of Effelsberg, Metsahovi, Shanghai, Arecibo (again), TIGO

5. 2008 May 30 EXPReS Annual Review: SA1Slide #5 e-VLBI: data rate improvements

6. 2008 May 30 EXPReS Annual Review: SA1Slide #6 Towards global e-VLBI Long-haul high-bandwidth data transport: TCP Reno on old linux kernels clearly inadequate Modern TCP stacks much better, UDP even better, but impossible without modification of Mark5A control code And can be hostile to other users on open network Stability of original Mark5 code a serious issue

7. 2008 May 30 EXPReS Annual Review: SA1Slide #7 Global e-VLBI: hacking time With several high-profile demos coming up: time to take matters in our own hands First step: re-enable UDP in Mark5A code abysmal performance... Incorporate VLBI-UDP in Mark5 code, in collaboration with colleagues from UMan Simplified version developed at JIVE, used in both APAN and EXPReS-Oz demo Complete re-write of Mark5 control software, with rigorous thread control, enabling packet dropping, re-ordering, padding Used successfully to transfer data at near- line-speed from European telescopes, from Arecibo and, recently, from South Africa and South America Large development effort, but has paid off in a big way

8. 2008 May 30 EXPReS Annual Review: SA1Slide #8 The road to China: the demo at Xi’An

9. 2008 May 30 EXPReS Annual Review: SA1Slide #9 The APAN demo Needed: Reliable transport from standard Mark5 at Shanghai (ancient Linux kernel) Physical connection between two non- communicating Chinese NRENs Some way to bypass the Great Chinese Firewall Interface between correlator and Australian telescope control systems On-the-fly translation from LBA to Mark5B format What could possibly go wrong......

10. 2008 May 30 EXPReS Annual Review: SA1Slide #10 APAN results Opened up Siberia connection Excellent PR Longest (e) baseline ever

11. 2008 May 30 EXPReS Annual Review: SA1Slide #11 The EXPReS-Oz demo Three Australian telescopes (Mopra, ATCA, Parkes) 12 hours at 512 Mbps without a hitch Beautiful detection of SN1987a

12. 2008 May 30 EXPReS Annual Review: SA1Slide #12 Correlator upgrades Network upgrade: All internal JIVE and e-VLBI traffic now through own routers (HP5412zl and HP3500yl-48 ProCurve) Found quite a few of (serious) bugs in HP equipment So far, one bug fix out of 5 active cases Multiple 1-Gbps UDP streams not foreseen by HP engineers? Dedicated network monitoring platform alongside routers Removal of dependencies on ASTRON network Upgrade of Mark5 hardware (motherboards, memory, power supplies) completed New data acquisition platform (Solaris AMD server, fully interchangeable with redundant control computers

13. 2008 May 30 EXPReS Annual Review: SA1Slide #13 New modes, feedback and streamlining Completed: General speedup of correlator control code Real-time Mark5 debug info feedback to stations Re-connecting of individual stations without job re-start Space-craft tracking mode Use of Westerbork synthesis data to improve e-VLBI calibration and target selection Network monitoring tools Currently under development: Adaptive observing On-the-fly fringe fitting Real-time station-log processing, display of Tsys, status information Mixed-mode e-VLBI Integration of Mark5B 1024 Mbps e-VLBI (and more..)

14. 2008 May 30 EXPReS Annual Review: SA1Slide #14 New network monitoring tools...

15. 2008 May 30 EXPReS Annual Review: SA1Slide #15 And new telescopes...

16. 2008 May 30 EXPReS Annual Review: SA1Slide #16 In summary Guaranteed high bandwidth through use of an aggressive protocol *and* reliable software Lightpath technology offers clear advantages; routed connections need negotiations Convincingly demonstrated that it is possible to overcome long-haul connectivity problems Simplicity paid off: trivial to connect new EVN stations once they come online

17. 2008 May 30 EXPReS Annual Review: SA1Slide #17

18. 2008 May 30 EXPReS Annual Review: SA1Slide #18

19. 2008 May 30 EXPReS Annual Review: SA1Slide #19 SA1: state of deliverables A total of 15 deliverables were due by month 24, 11 at JIVE and 4 at the University of Manchester 8 out of 11 JIVE deliverables were completed Most of the work on the remaining 3 was finished 3 deliverables that were not due until the last year of EXPReS were completed well ahead of time The 4 UniMan deliverables were delayed because of the delay in the delivery of the e-Merlin correlator. Most of the IBOB design work has been completed, but tests await the arrival of the first I/O boards, in the summer of 2008

20. 2008 May 30 EXPReS Annual Review: SA1Slide #20 Outsourcing Three workpackages were outsourced to MIT: SA1.24: Mark5A code modifications Working version available (though no “official” release yet) SA1.25: Mark5B code modifications Dimino (sending) side largely done in-house Domino (receiving) side developed at MIT, debugging in progress Extensive engineering support in integration of Mark5B SA1.20: Improved network applications mainly done in-house, with MIT support

21. 2008 May 30 EXPReS Annual Review: SA1Slide #21 SA1: the next 18 months The focus of SA1 for the next 18 months will be on Adaptive scheduling Further integration of Mark5B in EVN operations Streamlining operational data rate adjustments Further improving the robustness and flexibility of the correlation process Mixed-mode operations Increasing the data rate beyond 1 Gbps

22. 2008 May 30 EXPReS Annual Review: SA1Slide #22 Specific tasks Fast/adaptive (re-)scheduling Transform EVN into a truly flexible instrument Based on automated download and execution of modified observing schedules - possibly on-the-fly Reactions from stations vary from “excellent idea” to “over our dead bodies” But will further improve reliability and responsiveness of EVN All bits tested and in place, full operational test is planned

23. 2008 May 30 EXPReS Annual Review: SA1Slide #23 Specific tasks (2) Data rate adjustments Packet dropping: Drop packets at sending side (while preserving headers!) Pad datastream at receiving side with dummy data Channel dropping: Selectively drop frequency channels at receiving side High demand on cpu, need hyperthreading, right kernels, dual cpu’s Mixed configurations: Use different configurations at different stations (limited to 32, 64, 128 etc Mbps)

24. 2008 May 30 EXPReS Annual Review: SA1Slide #24 Specific tasks (3) Mixed-mode operations: Enable all possible combinations of recording (at stations and/or at correlator), real-time and delayed data transfer Make best use of complete EVN network, including stations that have less-than optimal connectivity

25. 2008 May 30 EXPReS Annual Review: SA1Slide #25 Specific tasks (4) Beyond 1 Gbps 1024 Mbps will simply not fit on 1 Gbps... Westerbork: use of DWDM equipment, dual 1-Gbps connection, channel bonding Could be used to Jodrell Bank as well (dual 1-Gbps lightpaths to JIVE) Onsala, Effelsberg: share 10-Gbps connections with eLOFAR, via VLAN (nearly in place) Next, Torun? Most sensitive telescopes in Europe soon at a full 1024 Mbps And tests ongoing at 4 Gbps, from Onsala

26. 2008 May 30 EXPReS Annual Review: SA1Slide #26 And soon to come... Yebes first light (any moment now) Hartebeesthoek: demo last month at 64 Mbps, S-African universities expect to be connected to Europe at 10 Gbps in 2009 Arecibo: recent tests at ~512 Mbps St Croix? Urumqi: 622 Mbps connection (still in rumor phase) Japan: talks ongoing about combining VERA telescopes with EVN in e-VLBI pilot project Australia-China-Japan e-VLBI demo upcoming (at e-VLBI workshop in Shanghai)

28. 2008 May 30 EXPReS Annual Review: SA1Slide #28 SA1 Deliverables D#AAD#DescripPlanActualStatus 11SA1.02Job preparation utilities16 4 27SA1.04eMERLIN VSI interfaces design 13)222 28SA1.05Network Protocol decision:16204 45SA1.09Tests using local Jodrell Bank home e-MERLIN telescope 14)22? 51SA1.13Tested software for operational improvements18 4 52SA1.14Test using remote e-MERLIN telescope 17)24? 54SA1.15VSI Interfaces 16)23164 71SA1.17Flexible local GE network 4)18194 84SA1.18Network monitoring tools 20)30244 85SA1.19Multiple e-MERLIN telescope tests24? 95SA1.2Improved network applications 3)172 96SA1.21Monitoring user interfaces3074 124SA1.22Mark5 hardware upgrade16204 126SA1.23Use of WSRT synthesis data for e-VLBI calibration17224 127SA1.24Mark5A code modifications172 128SA1.25Mark5B code modifications172 136SA1.26Investigating a 1024M sub-array20234 148SA1.27Space craft tracking correlator mode27204 161SA1.31Removing/adding stations from the correlation process on the fly34204 SA1- Past 12 month overview