1. Jason Manley, Aaron Parsons, Don Backer, Henry Chen, Terry Filiba, David MacMahon, Peter McMahon, Arash Parsa, Andrew Siemion, Dan Werthimer, Mel Wright

2. Outline What is a correlator? Scalable packetized correlators: – The architecture – The hardware – The software – The cost Closing thoughts Walk through actual design Questions and comments

3. Interferometry…

5. Basic idea Amplitude Time 90°  V ij  V ij ViVi VjVj ∑ ∑ Z -n 90°  V ij  V ij ViVi VjVj ∑ ∑ Z -n 90°  V ik  V ik VkVk ∑ ∑  V ii ∑  V jj ∑  V jk ∑  V jk ∑  V kk ∑ Amplitude Time

6. “Actual” FX Correlator ∑ FFTZ -n FFTZ -n FFTZ -n ∑∑

7. CASPER DSP backend concept

8. Design Philosophy Standardized processing hardware Commercial interconnect Asynchronous compute engines Synchronization using common 1PPS UDP output delivery over ethernet network Correlator scales with your array

9. CASPER FX Architecture

10. Implementation

11. Architecture to hardware mapping Example 8 Antenna system

12. F Engine Operations Reformat DDCQuantize Channelize ADC Two F engines per iBOB Dual polarization design Currently uses ASTRO library Currently processes data at native clock rate (<200MHz IBOB or < 400MHz ROACH)

13. Setup and Control Clocks: – X engines each run off independent clock – Sampling synchronized at F engines, but clock not distributed to X engines Synchronized using global 1pps signal at ADCs – Propagated to X engines using out-of-band signaling on XAUI links – Headers labeling 10GbE Ethernet packet data System control: separate 100Mbps Ethernet network on BEE2 F engines configured from BEEs through XAUI links Control packets: CASPER UDP framework on BEE2 control FPGA Execute Python scripts for configuration, control and debugging

14. F engine development 2008: – Coarse delays (cable length compensation) – Fringe-stopping & fine delays – Walsh code generation and phase switching – Real sampling (low bandwidth) – Parallel streams (high bandwidth) Future: – Ability to output subset of band – Spectral zoom modes

15. X Engine Operations Using CASPER library Scales with 2^N antennas Fit as many X engines on an FPGA as possible (2x 16 ant on BEE2 usr) 10GbEBufferX EngAccum

16. Backend Software UDP packets received Currently received, parsed and saved in MIRIAD file format by single computer. Computing requirements dependant on experiment; Usually single computer ok: 128 antennas, 1 sec integrations, 2k chan = 512MB/s

17. Pending systems Bench sys: 8ant, DP, 200MHz, 2k ch PAPER: 128ant, DP, 100MHz, 2k ch KAT-7: 8ant, DP, 256MHz, 2k ch meerKAT: 80ant, DP, 1GHz, 16k ch Bologna: 32ant, SP, 32MHz, 1k ch GMRT: 32ant, DP, 400MHz, 4k-8k ch

18. How does it scale

19. FPGA Roadmap Processing power doubling every two years V4 = ½ power requirements of V2Pro* * Manufacturers claim - Xilinx Inc.

20. Coming soon… 10Gbps output optionally gives integrations ~10ms More efficient use of hardware DSP slices High speed, scalable, distributed data capture software Walsh codes and phase switching Phase rotation 64 antenna design Upgrade to 4096 channels ROACH hardware: – <400MHz bandwidth – 16 384 channels – 128 antennas – no architectural changes

21. Questions and Comments Visit the CASPER correlator page: http://casper.berkeley.edu/wiki/index.php?title=Correlator Add your own requirements: http://casper.berkeley.edu/wiki/index.php?title=International_ Correlator_Collaboration Email me: jason_manley@hotmail.com

23. PFB-FFT response

24. Current uses Pocket Spectrometer Using ATMEL ADC’s at 2 Gsamples/sec Performing 4 real FFT’s in 1 (complex) biplex pipelined FFT module. 2048 channels Uses just 1 ADC, 1 IBOB, and your laptop.

25. ROACH block diagram