ngVLA Correlator with 2015ish Technology Jack Hickish, Dan Werthimer, and CASPER Collaboration

CASPER Correlator Collaboration CASPER Correlator Collaboration PAPER, HERA (FPGA F, GPU X) PAPER, HERA (FPGA F, GPU X) MeerKAT (FPGA F, FGPA X) MeerKAT (FPGA F, FGPA X) AMI AMI GMRT GMRT Bologna Bologna ISI (Infrared) ISI (Infrared) SKADS (Oxford) SKADS (Oxford) SMA (CFA, ASIAA) SMA (CFA, ASIAA) MIT FFT direct imaging correlator MIT FFT direct imaging correlator FASR FASR Baryon Acoustic Oscillation experiments Baryon Acoustic Oscillation experiments LEDA (EOR) LEDA (EOR) Allen Telescope Array (90 uS imaging) Allen Telescope Array (90 uS imaging)

Berkeley Correlator Team Berkeley Correlator Team Dan Werthimer (28 years correlator design)Dan Werthimer (28 years correlator design) Matt Dexter (20 years correlator design)Matt Dexter (20 years correlator design) David McMahon (10 years correlator design)David McMahon (10 years correlator design) Aaron Parsons (6 years correlator design)Aaron Parsons (6 years correlator design) Rick Raffanti (ADC and RF/analog board designer – 30 years)Rick Raffanti (ADC and RF/analog board designer – 30 years) Dave Deboer (project manager)Dave Deboer (project manager) Andrew Siemion (Post Doc) – correlators, transient, pulsars, SETIAndrew Siemion (Post Doc) – correlators, transient, pulsars, SETI Hong Chen (Astr. Undergrad – parameterized FPGA designs)Hong Chen (Astr. Undergrad – parameterized FPGA designs) Jack Hickish – Cambridge/UCB Postdoc – HERA correlatorJack Hickish – Cambridge/UCB Postdoc – HERA correlator Stevo Bailey – ASIC Design – Hurricane CorrelatorStevo Bailey – ASIC Design – Hurricane Correlator 3

HERA Array 350 x 15 meter dishes HERA Array 350 x 15 meter dishes

Simultaneous Digital Backends Piggyback, Commensal, Sky Surveys Analog Power Splitters Analog Power Splitters or or Digital Data Splitter Digital Data Splitter

Beowulf Cluster Like General Purpose Architechture Dynamic Allocation of Resources, need not be FPGA based

FX Correlator FX Correlator

Antenna 2 F engine Antenna 3 F engine Frequency Band 1 X engine Frequency Band 2 X engine Antenna 1 F engine Frequency Band 3 X engine 8

CASPER FXB Correlator/Beamformer (correlator needed to calibrate beamformer)

Correlators and Beamformers Correlators and Beamformers Globally Asynchronous (like a computer cluster)Globally Asynchronous (like a computer cluster) Data is time stamped with 1 PPS at ADCData is time stamped with 1 PPS at ADC Locally Synchronous, Globally AsynchronousLocally Synchronous, Globally Asynchronous Solve problem of correlator/beamformer interconnect problem by using 10/40 Gbe switches (for both interconnect and fast readout) Solve problem of correlator/beamformer interconnect problem by using 10/40 Gbe switches (for both interconnect and fast readout) No need for high density complex boardsNo need for high density complex boards 10

CASPER FPGA Packetized Correlator CASPER FPGA Packetized Correlator

Packetized FX Correlator Packetized FX Correlator

Heterogeneous Correlator Heterogeneous Correlator

ngVLA Correlator Specs ? ngVLA Correlator Specs ? 256 antenna dual pol (512 antpols) 256 antenna dual pol (512 antpols) 50 GHz bandwidth (100 Gsps) 50 GHz bandwidth (100 Gsps) 2015 technology and pricing 2015 technology and pricing (cost, complexity, and power consumption (cost, complexity, and power consumption will decrease) will decrease)

ADC’s 26 Gsps 3.5 bit Hittite Dave Hawkins: board and Altera interface Rick Raffanti: FMC board and Xilinx Interface see CASPER Memo Gsps 8 bit Fujitsu (die) 32 Gsps 6 bit Micram 80 Gsps 8 bit Keysight 80 Gsps 8 bit Berkeley (freq) 80 Gsps 8 bit Berkeley (time) ADC’s 26 Gsps 3.5 bit Hittite Dave Hawkins: board and Altera interface Rick Raffanti: FMC board and Xilinx Interface see CASPER Memo Gsps 8 bit Fujitsu (die) 32 Gsps 6 bit Micram 80 Gsps 8 bit Keysight 80 Gsps 8 bit Berkeley (freq) 80 Gsps 8 bit Berkeley (time)

Jack Hickish ngVLA Design Jack Hickish ngVLA Design Hittite 26 Gsps ADC (four ADC’s per antpol)Hittite 26 Gsps ADC (four ADC’s per antpol) NetFPGA-SUME boards for both F and XNetFPGA-SUME boards for both F and X Mellanox SX Gbe switchesMellanox SX Gbe switches (ten 5 GHz correlators) (ten 5 GHz correlators) $30M$30M

NetFGPA Board Virtex7 XC7V690T NetFGPA Board Virtex7 XC7V690T

ngVLA correlator 2015 cost $28M ngVLA correlator 2015 cost $28M ADC’s boards 2048 x $3K/hittite $6M ADC’s boards 2048 x $3K/hittite $6M F engine boards: 512 x XCKU11P $2M F engine boards: 512 x XCKU11P $2M X engine boards: 1200 x XCVU13 $4M X engine boards: 1200 x XCVU13 $4M Switches ( port QSFP) $5M Switches ( port QSFP) $5M 51,200 DWDM transceivers (80 km) $5M 51,200 DWDM transceivers (80 km) $5M racks, power, UPS, cables, cooling $1M racks, power, UPS, cables, cooling $1M labor (depends on Nmodes) $5M labor (depends on Nmodes) $5M

2015 Power Consumption, Size 2015 Power Consumption, Size F engines (at the antenna) 60KW F engines (at the antenna) 60KW (includes ADCs, FPGA boards, transceivers) (includes ADCs, FPGA boards, transceivers) (about 250 watts per telescope) (about 250 watts per telescope) X engine 170KW X engine 170KW (120KW/FGPA boards, 50KW/switches) (120KW/FGPA boards, 50KW/switches) 12 racks – 10U per 1GHz correlator 12 racks – 10U per 1GHz correlator (4U switch, 6U FPGA boards) (4U switch, 6U FPGA boards)

21 lags 300kHz clock discrete transistors $19, – First Radio Astronomy Digital Correlator Sandy Weinreb

Correlator processing power DLB DXB VLA GFlops 1 DCB LOFAR SMA DAS EVN/WSRT ALMA SKA. EVLA source: Arnold van Ardenne

Ray Escoffier “With correlator performance having gone up by a factor of 922,000 over the last 30 years, its only fair that correlator design engineers' salaries should have gone up by a similar factor!!”