1. SCARIe: eVLBI Software Correlation Over Dynamic Lambda Grids
Damien Marchal, on behalf of the SCARIe team.
University of Amsterdam
The Netherlands

2. SCARIe, is a project to make a software correlator for VLBI on top of a grid middleware and infrastructures.

3. Software Correlation Over Dynamic Lambda Grids
Outline
Introduction:
Radio astronomy
VLBI and eVLBI
Correlation
The SCARIe project
What is a software correlator
Two mode of operation: Batch or Real-time
Networking challenges
Experiment using DAS-3 and StarPlane
Conclusions
03/01/2019
Software Correlation Over Dynamic Lambda Grids

4. Software Correlation Over Dynamic Lambda Grids
Outline
Introduction:
Radio astronomy
VLBI and eVLBI
Correlation
SCARIe project
What is a software correlator
Two mode of operation: Batch or Real-time
Networking challenges
Experiment using DAS-3 and StarPlane
Conclusions
03/01/2019
Software Correlation Over Dynamic Lambda Grids

5. Software Correlation Over Dynamic Lambda Grids
Radio and optical
408 Mhz
optical
1.4 Ghz
03/01/2019
Software Correlation Over Dynamic Lambda Grids

6. Larger dish: higher resolution
λ / b
Sensitivity:
b2
λ = wavelength
b = diameter telescope
Arecibo: D = 305 m
03/01/2019
Software Correlation Over Dynamic Lambda Grids

7. Aperture Synthesis Imaging
A technique that uses a number of telescopes to simulate a much larger one. A larger dish, real or simulated, improves image clarity and brightness. This requires coordination between the telescopes and a supercomputer. Consider the examples displaying aperture size, aperture distribution and image quality.
Image Credits: Avruch and Pogrebenko
03/01/2019
Software Correlation Over Dynamic Lambda Grids

8. Westerbork / Very Large Array
03/01/2019
Software Correlation Over Dynamic Lambda Grids

9. Software Correlation Over Dynamic Lambda Grids
Even higher resolution: VLBI
03/01/2019
Software Correlation Over Dynamic Lambda Grids

10. Radio-Telescopes participating…
03/01/2019
Software Correlation Over Dynamic Lambda Grids

11. Software Correlation Over Dynamic Lambda Grids
Data Flow
Past
Today
Results
Today
Soon
Today
03/01/2019
Software Correlation Over Dynamic Lambda Grids

12. Software Correlation Over Dynamic Lambda Grids
Data Flow
Past
Today
Results
Today
Soon
Today
correlation
03/01/2019
Software Correlation Over Dynamic Lambda Grids

13. Software Correlation Over Dynamic Lambda Grids
Outline
Introduction:
Radio astronomy
VLBI and eVLBI
Correlation
SCARIe project
What is a software correlator ?
Two mode of operation: Batch or Real-time
Networking challenges
Experiment using DAS-3 and StarPlane
Conclusions
03/01/2019
Software Correlation Over Dynamic Lambda Grids

14. Software Correlation Over Dynamic Lambda Grids
SCARIe: the project
The starting point:
The increase of size/performance of a modern grids/clusters and networks make of grid a “to investigate” plate-form for software correlation.
Today
Results
Software correlator
(on a grid)
Today
03/01/2019
Software Correlation Over Dynamic Lambda Grids

15. Software Correlation Over Dynamic Lambda Grids
SCARIe: the project
The starting point:
The increase of size/performance of a modern grid/cluster and networks make of grid a “to investigate” plate-form for software correlation.
Software Correlator
High flexibility:
Tracking of spacecrafts
Research on masers
Share the resource/grid with others to lower the cost
Run everywhere
Hardware correlator
Low flexibility
Superior computing power
More efficient use of computing resources (Lower energy consumption)
03/01/2019
Software Correlation Over Dynamic Lambda Grids

16. Software Correlation Over Dynamic Lambda Grids
SCARIe: distributed correlation
Telescopes
Correlator
03/01/2019
Software Correlation Over Dynamic Lambda Grids

17. Software Correlation Over Dynamic Lambda Grids
SCARIe: distributed correlation 
Telescopes
Correlator
Input nodes
03/01/2019
Software Correlation Over Dynamic Lambda Grids

18. Software Correlation Over Dynamic Lambda Grids
SCARIe: distributed correlation 
Telescopes
Correlator
Input nodes
Correlation nodes
03/01/2019
Software Correlation Over Dynamic Lambda Grids

19. Software Correlation Over Dynamic Lambda Grids
SCARIe: distributed correlation
Telescopes
Correlator
Input nodes
Input flow = sum(output flows)
Correlation nodes
Output node
03/01/2019
Software Correlation Over Dynamic Lambda Grids

20. Software Correlation Over Dynamic Lambda Grids
SCARIe: distributed correlation 
Telescopes
Correlator     
Input nodes
Input flow = sum(output flows)‏
Current experiment at:
4 x 256Mbps
Full size experiment:
16x1Gbps
Future eVLBI:
32x4Gbps
Correlation nodes
Output node
03/01/2019
Software Correlation Over Dynamic Lambda Grids

21. SCARIe: distributed correlation
We can identify two kinds of data flows.
Input flows, link to the RT.
Correlator
In-grid flows.
03/01/2019
Software Correlation Over Dynamic Lambda Grids

22. SCARIe: distributed correlation
We can identify two kinds of data flows.
Input flows, link to the Radio-Tel.
(point to point, long distance, several Gbps)
Correlator
In-grid flows.
(one to all, short distance, low latency).
03/01/2019
Software Correlation Over Dynamic Lambda Grids

23. SCARIe: operational mode
Two kinds of operational mode
Batch correlation:
the data are saved on hard-drives and replayed;
the network performance and number of nodes impact the processing speed.
Real-time correlation:
the radio-telescopes are streaming the data directly into the software correlator;
the network performance and number of nodes impact the success of the observation.
03/01/2019
Software Correlation Over Dynamic Lambda Grids

24. SCARIe: operational mode
We can identify two kind of operational mode…
Batch correlation:
the data are saved on hard-drives and replayed;
the network performance and number of nodes impact the processing speed.
Fastness !
Real-time correlation:
the radio-telescope are streaming the data directly into the software correlator;
The network performance and number of nodes impact the success of the observation.
Reliability!
03/01/2019
Software Correlation Over Dynamic Lambda Grids

25. Software Correlation Over Dynamic Lambda Grids
SCARIe: workflow
03/01/2019
Software Correlation Over Dynamic Lambda Grids

26. Software Correlation Over Dynamic Lambda Grids
SCARIe: workflow
03/01/2019
Software Correlation Over Dynamic Lambda Grids

27. Software Correlation Over Dynamic Lambda Grids
SCARIe: workflow
03/01/2019
Software Correlation Over Dynamic Lambda Grids

28. Software Correlation Over Dynamic Lambda Grids
Outline
Introduction:
Radio astronomy
VLBI and e-VLBI
Correlation
SCARIe project
What is a software correlator
Two mode of operation: Batch or Real-time
Networking challenges
Experiment using DAS-3 and StarPlane
Conclusions
03/01/2019
Software Correlation Over Dynamic Lambda Grids

29. Software Correlation Over Dynamic Lambda Grids
Experiment on DAS-3
Goals:
evaluate the scalability of a software correlator;
evaluate the DAS-3 network capabilities;
evaluate Bandwidth on Demand service.
03/01/2019
Software Correlation Over Dynamic Lambda Grids

30. Software Correlation Over Dynamic Lambda Grids
What is DAS-3 ?
DAS-3 is composed of:
5 cluster sites,
1GE Network,
10G Myrinet network,
Interconnexion photonic network called StarPlane.
StarPlane manages:
several 10 Gbps lighpaths
lightpaths have to reserved
03/01/2019
Software Correlation Over Dynamic Lambda Grids

31. Software Correlation Over Dynamic Lambda Grids
What is StarPlane ?
Key principle: applications have to request to StarPlane the right to use a certain amount of lightpaths of the fast interconnexion network.
Two ways to acquire a path with StarPlane:
procedurally by direct calling the webservice function: getPath(“source”, “dst”, “time”, “duration”)
declaratively through a query language like:
query( path(X, “leiden”, L1),
path(X, “delft”, L2), available([L1, L2]), reserve([L1, L2], ResTicket) )‏ X L1 L2
leiden
delft
03/01/2019
Software Correlation Over Dynamic Lambda Grids

32. Software Correlation Over Dynamic Lambda Grids
DAS-3 + StarPlane
Demonstration: Two applications are transferring data between two clusters. Initially they start using the default Internet interconnexion network. The middleware requests path to StarPlane. When path is acquired the middleware redirect the traffic to make use of it.
03/01/2019
Software Correlation Over Dynamic Lambda Grids

33. Software Correlation Over Dynamic Lambda Grids
DAS-3 + StarPlane
Demonstration: Two applications are transferring data between two clusters. Initially they start using the default Internet interconnexion network. The middleware requests path to StarPlane. When path is acquired the middleware redirect the traffic to make use of it.
'a' is started and uses the shared 1GE network;
'b' is started and and uses the shared 1GE network;
'a' has acquired the right to use one of the lighpath.
03/01/2019
Software Correlation Over Dynamic Lambda Grids

34. SCARIe experiment on DAS-3
Correlator
03/01/2019
Software Correlation Over Dynamic Lambda Grids

35. SCARIe experiment on DAS-3
Mapping the application to the resource/service set:
Correlator
03/01/2019
Software Correlation Over Dynamic Lambda Grids

36. SCARIe experiment on DAS-3
Mapping the application to the resource/service set:
Correlator
03/01/2019
Software Correlation Over Dynamic Lambda Grids

37. SCARIe experiment on DAS-3
Mapping the application to the resource/service set:
Correlator
03/01/2019
Software Correlation Over Dynamic Lambda Grids

38. SCARIe experiment on DAS-3
Node and network resource co-allocation query:
query(
cluster(“UvA”),
cluster(C2), cluster(C3), C2 \= C3,
NC1 + NC2 + NC3 == 100,
nodes(“UvA”, NC1, N1),
nodes(C2, NC2, N2),
nodes(C3, NC3, N3),
path(“UvA”, C2, L1),
path(“UvA”, C3, L2),
reserve([N1, N2, N3, L1, L2], ResTicket) )‏
03/01/2019
Software Correlation Over Dynamic Lambda Grids

39. Software Correlation Over Dynamic Lambda Grids
Results
03/01/2019
Software Correlation Over Dynamic Lambda Grids

40. Software Correlation Over Dynamic Lambda Grids
Results
Input nodes (quad core 2Ghz CPU) are close to handle 1Gbps data flow.
We need around 100 correlation nodes to compute a 4x256Mbps observation in real-time. This is much more than our “theoretical estimation”.
Networking speed is not (yet ?) a bottleneck.
StarPlane permits a simple, per application QoS between the clusters by controlling the access to the medium at a large grain.
03/01/2019
Software Correlation Over Dynamic Lambda Grids

41. Software Correlation Over Dynamic Lambda Grids
Future plan
We need to use a BoD service (like GN2 Autobahn) to establish QoS paths between the radio-telescopes and the grid.
Correlator
International Bandwidth on Demand Service (eg. Gn2 AutoBahn)
Last 200m
03/01/2019
Software Correlation Over Dynamic Lambda Grids

42. Software Correlation Over Dynamic Lambda Grids
Future plan
Integrate the BoD service into the “default” StarPlane network service to submit complex, integrated, jobs to DAS-3.
query(
path(CINPUT, L1),
path(CINPUT, L2),
cluster(CINPUT), cluster(C2), cluster(C3),
alldifferent( [CINPUT, C2, C3] ),
NC1 + NC2 + NC3 == 100,
NC1 >= NumRT,
nodes(CINPUT, NC1, N1),
nodes(C2, NC2, N2),
nodes(C3, NC3, N3),
path(CINPUT, C2, L1),
path(CINPUT, C3, L2),
reserve([N1, N2, N3, L1, L2], ResTicket) )‏
03/01/2019
Software Correlation Over Dynamic Lambda Grids

43. Software Correlation Over Dynamic Lambda Grids
Conclusion
Software correlation is challenging :
lots of input flows to bring the data;
lots of computations that need to be distributed;
over multi-domain;
for large size (Gbps speed);
over several scales: LAN, nation, world;
with a “per experiment” dynamicity.
Software correlation is possible but it is unclear at what cost !
In term of computing power:
CPU is slow for a systematic computation like FFT.
=> using GPU/FPGA for co-processing.
In term of networking:
fat pipes have a cost;
dynamicity (connectivity) has a cost;
QoS has a cost.
=> Selecting the adequate layer to implement the needed service.
03/01/2019
Software Correlation Over Dynamic Lambda Grids

44. Software Correlation Over Dynamic Lambda Grids
Questions/Answers
Contact information
Nico Kruithof
Joint Institute for VLBI in Europe (JIVE)
Damien Marchal
Universiteit van Amsterdam (UvA)
SCARIe is made possible through the support of the Netherlands Organisation for Scientific Research (NWO).
03/01/2019
Software Correlation Over Dynamic Lambda Grids

45. Middleware for StarPlane
Based on linux dlopen to hijack the socket API.
Application
Application
Starplane
Hijacker
Hijacker
Socket API
Normal interconnexion
Socket API
Lighpath if available
03/01/2019
Software Correlation Over Dynamic Lambda Grids