1. The 7th International VLBI Technology Workshop Thailand
We acknowledge the Wajarri Yamatji people as the traditional owners of the Murchison Radio Observatory site.
The 7th International VLBI Technology Workshop
Thailand
ASKAP Correlator
CASS
John Tuthill, Mike Pilawa, Chris Phillips, Adam Macleod, Grant Hampson, John Bunton, Andrew Brown, Mia Baquiran, Daniel George, Tim Bateman
November 2018

2. Outline: ASKAP Instrument Overview FPGA-based Hardware Correlator
design
performance
Oversampled filterbanks, why?
ASKAP Extensions
Other research and projects
ASKAP Correlator | John Tuthill

3. ASKAP- Australian SKA Pathfinder
Area: 41,200 km2
Population: 120
(3 milli-persons per km2)
ASKAP Digital Beamformer | John Tuthill

4. ASKAP PAF 188 elements (94 elements in each polarisation)
Analogue RF bandwidth: 700MHz to 1.8GHz
Band 1: 700MHz – 1,200MHz
Band 2: 840MHz – 1,440MHz
Band 3: 1,400MHz – 1,800MHz
Field of View 30 deg2
Analogue RF over Fibre (RFoF)
Thermoelectric coolers
36 x 12m dia. dishes
Max baseline = 6 km
~3000m2 effective collecting area
ASKAP Correlator | John Tuthill

5. ASKAP: Architecture and signal chain
Custom digital system:
Raw data ingest: 130Tbits/s
Raw processing power: 2.3PMAC’s/s
Mk-II PAF: checkerboard array, LNAs, RF filters, laser drivers
Central Site DSP: FPGA Hardware
Correlator
Ethernet
Raw Visibilities
Optical-to-electrical,
Samplers,
Coarse filterbanks
Beamformers,
Fine filterbanks
RF over
Fibre
Imaging pipeline : iVEC “Galaxy”
Cray Inc. XC30 series supercomputer
Located in Perth, ~800km from MRO.
~200 TeraFLOPS
ASKAP Correlator | John Tuthill

6. ASKAP DSP Hardware Passive optical circuits for data cross-connects
Redback-3 platform
Dragonfly-3 Digital Receiver
Beamformer
Correlator
ASKAP Correlator | John Tuthill

7. ASKAP – Digital Receiver
Sampling
Resolution 12bits (9.4 ENOB)
Analog BW 2.8GHz
Direct digital down-conversion (2nd and 3rd Nyquist zones)
Sample rates: 1,280MSps and 1,536MSps
Coarse frequency channelization
Processed bandwidth: 384 MHz (arbitrarily selectable anywhere in the observing band)
Coarse frequency channels: 1MHz oversampled by 32/27
Spectral flatness: <0.2dB
Sub-band alias rejection: >60dB
Dragonfly-3 Digital Receiver
ASKAP Correlator | John Tuthill

8. ASKAP – Beamformer Digital beamforming Fine frequency channelization
Narrow-band beamformer structure (weight-and-sum)
Full 192x192 array covariance matrix for each 1MHz channel
Up to 72 single-polarized beams (usually configured as 36 dual-pol beams)
Fine frequency channelization
6 frequency “zoom” modes: 18.5kHz, 9.3kHz, 4.6kHz, 2.3kHz, 1.2kHz, 580Hz
Critically sampled
Delay tracking and fringe-stopping
Coarse delay to 1us resolution
Time-varying phase slope applied to fine channels
Redback-3 DSP Platform
ASKAP Correlator | John Tuthill

9. ASKAP Max SNR Beamforming
ack. Aidan Hotan
Signal + Noise
Noise
Signal - =
Array Covariance Matrix
Inverse
Dominant
Eigenvector = x
ASKAP Correlator | John Tuthill

10. ASKAP – Correlator Full Stokes correlator for
630 Baselines
36 dual-polarised beams
15,552 spectral channels
Same hardware as beamformer
(Different FPGA firmware)
5 second integrations
32-bit float output
~50Gbps over Ethernet
Visibilities streamed to Pawsey Centre HPC in Perth
ASKAP Correlator | John Tuthill

11. Oversampled filterbanks: critically sampled case, order = 1024
Channel spacing = Output sampling rate
“Scalloping” across the full-band
Significant aliasing!!!
Channel spacing = 1MHz
Output sample rate = 1MHz
Three output channels
Single output channel
with two of the images
ASKAP Correlator | John Tuthill

12. Oversampled filterbanks: Fractionally over-sampled case (8/7), order = 1024
Channel spacing < Output sampling rate
Almost no “Scalloping”
Aliasing reduced from -6dB to -50dB!!
Channel spacing = 1MHz
Output sample rate = MHz
Three output channels
Single output channel
with two of the images
ASKAP Correlator | John Tuthill

13. Oversampled filterbanks: Full-band response
Critically sampled
Over sampled
ASKAP Correlator | John Tuthill

14. Oversampled filterbanks: Notes
Allows independent control of channel response and aliasing characteristics
Increases the overall output data rate by the OS factor
Sample rates no longer conform to VDIF specification
Using a prototype “CODIF” format for Effelsberg and Jodrell Bank
CODIF - CSIRO Oversampled Data Interchange Format (provisional)
ASKAP Correlator | John Tuthill

15. Future work: ASKAP Tied-Array
Commensal operation
Uses existing FPGA-based hardware correlator
Can start small and scale up
Development compatible with FRB fast-dump correlator
ASKAP Correlator | John Tuthill

16. New tech.: Xilinx RFSoC (RF System-on-Chip)
One chip!!
12-bit RF-ADCs (+ DACs)
16 x 2GSps or 8 x 4GSps
4GHz analogue BW
Synchronous sampling across multiple devices
Processing system
Quad-core ARM Cortex-A53
Dual-core ARM Cortex-R5
Rich set of standard interfaces (DDR4, USB, ...)
FPGA fabric
~1 million logic cells
~4k hardware multipliers
Fast RAM
32.75Gbps transceivers
ASKAP Digital Beamformer | John Tuthill

17. RFSoC Projects: “Bluering” – currently in prototype HW development phase
Modular, scalable to 512 RF inputs (32 RFSoC devices)
RFoF inputs
Direct RF sampling
12-bits
SNR: 66 dB
Cross talk: -70 dBc
Array-based DSP
Beamforming
Correlation
Channelisation
Synchronous transient capture
Physical/Electrical
Overall size: 600x600mm (TBD)
Weight: 20kg
Supply: 48V 900W
Liquid cooled
Optical data transport
Off-system: 40GbE, SMF (40km)
Monitoring and Control: 10GbE SMF (40km)
ASKAP Correlator | John Tuthill

18. RFSoC Projects: CABB upgrade & Cryo-PAF
Prototyping CABB upgrade using the Xilinx ZCU111 RFSoC board
(very encouraging results so far!!)
Future: Multi-channel sampler and back-end processor for the Cryo-PAF project
ASKAP Digital Beamformer | John Tuthill

19. Other Projects Ultra-wideband feed system for Parkes 64m
CSIROSat-1: 3U CubeSat
Hyperspectral IR Earth imaging
On-board FPGA and SoC image processing
In-orbit re-programming
S-Band down-link
Technology demonstrator
Capability building
ASKAP Correlator | John Tuthill

20. Thank you Star-on Machine CSIRO Astronomy and Space Science
Dr. Seuss - The Sneetches and Other Stories
Thank you
CSIRO Astronomy and Space Science
Dr John Tuthill Research Group Leader | Signal Processing Technologies t e w
CSIRO Astronomy and space science

21. “CODIF” prototype
“CODIF” header
ASKAP Correlator | John Tuthill

22. Future work: ASKAP Tied-Array
Tied-Array Processor (TAP)
cell
ASKAP Correlator | John Tuthill

23. Detailed Signal Path – single PAF
Digital Receiver
Beamformer
Brief
Number representation is fixed-point signed two’s complement throughout
ASKAP Correlator | John Tuthill

24. Digital Beamformer (single DSP FPGA)
ASKAP Correlator | John Tuthill

25. Individual 1MHz digital beamformer module
ASKAP Correlator | John Tuthill

26. Beamformer engine performance
Two main performance bounds:
Degrees of freedom in beam weight selection (architecture)
Maximum beam weight update rate (interface)
1. Degrees of freedom (# active weights)
Sample rate = 1MHz x 32/27 -> T = ns
Beamformer logic clock = 312.5MHz
18 physical instances of the beamformer process ~1 port voltage per clock cycle
All contributing PAF port voltages must be processed in 263 clocks
18 single-pol beams: all 192 ports active
36 single-pol beams (1 replay): 130 active ports
72 single-pol beams (2 replays): 60 active ports
Implications for sidelobe control, null-steering, spill-over control
ASKAP Correlator | John Tuthill

27. Beamformer engine performance
2. Maximum weight update rate, determined by:
ACM dump time
Beam weight upload time
ACM data volume and dump time:
192 x 192 ports ÷ 2 (conjugate symmetry) x 2re/im x 32-bit float
≈172kBytes
Measured ACM dump time ≈ 4ms/ACM (172kB/4ms = 43MB/s = 344Mb/s)
(More efficient to stream raw port voltages than ACMs for dump times < 0.2ms)
Beam weight upload time ≈ 180ms/chassis (36 dual-pol beams, measured)
Max “dynamic” beamforming period ≈ 48 x 4ms + 180ms ≈ 400ms
(does not include off-line weight calculation)
Implications for real-time/adaptive beamforming and RFI tracking
ASKAP Correlator | John Tuthill

28. Beamforming Algorithm
Maximum SNR with phase matching
other approaches…
sub-space projection (RFI mitigation, see next presentation!)
Shape-constrained beamforming
(where )
weights modified for smooth phase:
Array covariance matrix + S W1 W2
W188
New beamformer weights
x 384 1MHz
frequency
channels
Brief
Dominant solution to the eigenvalue problem
ASKAP Correlator | John Tuthill