Correlators, Spectrometers, Beam Formers and VLBI using general purpose FPGA boards, tools & libraries (how to build eight radio astronomy instruments in two years) Dan Werthimer & Aaron Parsons University of California, Berkeley

Our research group is really 3 groups SETI (plus primordial black holes, HI mapping)SETI (plus primordial black holes, HI mapping) Public Participation Distributed ComputingPublic Participation Distributed Computing CASPER – Center for Astronomy Signal Processing and Electronics ResearchCASPER – Center for Astronomy Signal Processing and Electronics Research

SETI Group David Anderson, Bob Bankay, Court Cannick, David Anderson, Bob Bankay, Court Cannick, Jeff Cobb, Kevin Douglas, Josh Von Korff, Jeff Cobb, Kevin Douglas, Josh Von Korff, Eric Korpela, Matt Lebofsky, Dan Werthimer

UC Berkeley SETI Programs Name Time Scale Search Type SERENDIPseconds radio sky survey mS - seconds radio sky survey Astropulse nS - mS radio sky survey SEVENDIPnS visible targetted SPOCK 1000 seconds visible targetted DYSON IR targetted

Public Participation Supercomputing Group David Anderson, Rom Walton, SETI Group David Anderson, Rom Walton, SETI Group aka Distributed Computingaka Distributed Computing aka “edge resource aggregation”)aka “edge resource aggregation”)

The Client

Statistics TOTAL RATE

BOINC: NSF Berkeley Open Infrastructure for Network ComputingBerkeley Open Infrastructure for Network Computing –General-purpose distributed computing framework. –Open source. –Will make distributed computing accessible to those who need it. (Starting from scratch is hard!)

Projects AstronomyAstronomy (Berkeley) –Astropulse (Berkeley) gravitational pulsar search (Caltech,…) –PlanetQuest (SETI Institute) (Berkeley, Univ. Washinton,…) Earth scienceEarth science –Climateprediction.net (Oxford) Biology/MedicineBiology/Medicine (Stanford, Scripts) –FightAIDSathome: virtual drug discovery PhysicsPhysics (Cern) OtherOther –Web indexing/search –Internet Resource mapping (UC Berkeley)

Where's the computing power? ● 2010: 1 billion Internet-connected PCs ● 55% privately owned ● If 100M participate: – 100 PetaFLOPs, 1 Exabyte (10^18) storage your computers academic business home PCs

CASPER: Center for Radio Astronomy Signal Processing and Electronics Research Henry Chen, Daniel Chapman, Pat Crescini, Christina DeJesus, Pierre Droz Kirsten Meder, Jeff Mock, Aaron Parsons, Andrew Siemion, Dan Werthimer Radio Astronomy Lab Don Backer, Paul Demorest, Matt Dexter, Carl Heiles, David McMahon, Mel Wright, Lynn Urry Berkeley Wireless Research Center Bob Broderson, Chen Chang, John Wawrzynek SETI Institute Dave Deboer

Casper Origins Casper Origins NSF proposal to build SETI spectrometer (2003)NSF proposal to build SETI spectrometer (2003) (added one paragraph: BTW, this can be used for other astronomy instrumentation, potential spin offs are ….) (added one paragraph: BTW, this can be used for other astronomy instrumentation, potential spin offs are ….) Reviewer’s comments (paraphased): ~”SETI is bullshit, SETI will never find anything, But these instruments are useful for the community, strongly recommend funding” But these instruments are useful for the community, strongly recommend funding”

CASPER Real-time Signal Processing Instrumentation (NSF ATI) Low NRE, shared by the communityLow NRE, shared by the community Rapid development (8 instruments / 2 years)Rapid development (8 instruments / 2 years) Open-source, collaborativeOpen-source, collaborative Reusable, platform-independent gatewareReusable, platform-independent gateware Modular, upgradeable hardwareModular, upgradeable hardware Industry standard communication protocolsIndustry standard communication protocols Low CostLow Cost

MOTIVATION ATA, SKA, Focal Plane Arrays, SETI, need >> PetaOp/sec Instruments take a long time to build, very high NRE

Allen Telescope Array 6.1-meter offset Gregorian (2.4-meter secondary) rim-supported, hydroformed dishes

ATA-42 Operational This Summer

The Radio Revolution

Inner core Station SKA Square Kilometer Array

The Problem with the Current Hardware Development Model Takes 5 yearsTakes 5 years Cost Dominated by NRE because of custom Boards, Backplanes, ProtocolsCost Dominated by NRE because of custom Boards, Backplanes, Protocols Antiquated by the time it’s released.Antiquated by the time it’s released.

Solution: Modular HardwareModular Hardware –Low number of board designs –Can be upgraded piecemeal or all together –Reusable –Standard signal processing model which is consistent between upgrades. is consistent between upgrades.

Solution: use FPGA’s 1 FPGA = 100 Pentium, 1/500 the power per op 3X improvement per year! Moores Law for FGPA’s

FPGA = Field Programmable Gate Array reconfigurable computing - 1 minute 100 times faster than CPU, 5 times less power integer arithmetic, not good at F. Point highly parallel (500 multipliers per chip) harder to program (mathlab simulink) tools to abstract the hardware away signal processing libraries

Compute Module Diagram

Platform-Independent, Parameterized Gateware What is Gateware?What is Gateware? –Design logic of FPGAs (between hardware and software) (between hardware and software) Need libraries for signal processing which don’t have to be rewritten every hardware generation.Need libraries for signal processing which don’t have to be rewritten every hardware generation. Matlab Simulink!Matlab Simulink!

Biplex Pipelined FFT Uses 1/6 the resources of the Xilinx module.Uses 1/6 the resources of the Xilinx module.

FFT controls (Verilog and Simulink Libraries) Transform lengthTransform length BandwidthBandwidth Complex or RealComplex or Real Number of PolarizationsNumber of Polarizations Input bit width and output bit widthInput bit width and output bit width twiddle coefficient bit widthtwiddle coefficient bit width Run-time programmable down-shiftingRun-time programmable down-shifting Decimate optionDecimate option

Filter Response: PFB vs. FFT

Additional PFB controls Filter overlapFilter overlap Width of filter coefficientsWidth of filter coefficients Window function for filter (hamming, hanning, etc.)Window function for filter (hamming, hanning, etc.) Import filter coefficients for custom filter performanceImport filter coefficients for custom filter performance Both FFT and PFB available as Verilog modules (no proprietary software, but not as portable between chips/architectures).

Digital Down-Converter Selectable # of FIR tapsSelectable # of FIR taps On-the-fly programmable mix frequencyOn-the-fly programmable mix frequency Selectable FIR coeffSelectable FIR coeff Agile sub-band selection.Agile sub-band selection.

X-Engine Correlation Architecture (Lynn Urry, Aaron Parsons)

X-Engine Architecture: applied to an arbitrary sized antenna array

Hardware and Software Libraries legend:

Applications

Global Interconnects Commercial Infiniband switch from Mellanox, Voltaire, etc.Commercial Infiniband switch from Mellanox, Voltaire, etc. –Packet switched, non- blocking –24 ~ 144 ports (4X) per chassis –Up to 10,000 ports in a system –200~1000 ns switch latency –400~1200 ns FPGA to FPGA latency –480Gbps ~ 2.88Tbps full duplex constant cross section bandwidth –<$400 per port

Beowulf Cluster Like General Purpose Architechture Dynamic Allocation of Resources, need not be FPGA based

Targeted Applications Moderate to high-bandwidth problemsModerate to high-bandwidth problems –For low bandwidths, just use CPUs Lower to mid-scale computationLower to mid-scale computation –For very large applications (SKA), may be more cost effective to design ASICs Rapid DevelopmentRapid Development

Applications Applications VLBI Mark 5B data recorder - Haystack – 500 MHzVLBI Mark 5B data recorder - Haystack – 500 MHz VLBA and Beamforming - CfA, Bob Wilson, Jonathan Weintroub VLBA and Beamforming - CfA, Bob Wilson, Jonathan Weintroub SETI – Arecibo (UCB)SETI – Arecibo (UCB) ATA (UCB, Seti Institute) ATA (UCB, Seti Institute) JPL/UCB/SI DSN 20 GHz, 2pol (Preston, Gulkis, Levin, Jones) JPL/UCB/SI DSN 20 GHz, 2pol (Preston, Gulkis, Levin, Jones) Correlators and Imagers:Correlators and Imagers: ATA (Mel Wright) ATA (Mel Wright) Reionization Experiment (Backer/NRAO) Reionization Experiment (Backer/NRAO) Carma Next Gen (Dave Hawkins, Caltech) Carma Next Gen (Dave Hawkins, Caltech) SKA demonstrator South Africa (Justin Jonas) SKA demonstrator South Africa (Justin Jonas) MWAR, LWA – MIT, NRL MWAR, LWA – MIT, NRL

128 Million Channel SETI Spectrometer 200 MHz Bandwidth, 2 Hz resolution200 MHz Bandwidth, 2 Hz resolution

1 GHz bandwidth “Pocket Spectrometer” Using ATMEL ADC’s at 2 Gsamples/secUsing ATMEL ADC’s at 2 Gsamples/sec Performing 4 real FFT’s in 1 (complex) biplex pipelined FFT module.Performing 4 real FFT’s in 1 (complex) biplex pipelined FFT module channels2048 channels Uses just 1 ADC, 1 IBOB, and your laptop.Uses just 1 ADC, 1 IBOB, and your laptop.

Portable VLBI backend Interfaces to MARK 5B data recorderInterfaces to MARK 5B data recorder 500 MHz spectrum recorder.500 MHz spectrum recorder. (This makes 4 instruments in 1 year!)(This makes 4 instruments in 1 year!)

VLBI Mark 5B Front End 500 MHz BW, 32 channel filter bank VLBI Mark 5B Front End 500 MHz BW, 32 channel filter bank

Multi-Purpose FPGA-Based Spectrometer – Low Bandwidth Xilinx Virtex-II 6000 FPGA Xilinx Virtex-II 1000 FPGA 256 MB DRAM 200 Mhz ADC Compact PCI Backplane Software 200 Mhz ADC 200 Mhz ADC 200 Mhz ADC I I Q Q Pol. 1 Pol. 2 { { 200 Aux. I/O

SERENDIP V Spectrometer

SETI Applications JPL/UCB/SI DSN Sky Survey (20 GHz Bandwidth)JPL/UCB/SI DSN Sky Survey (20 GHz Bandwidth) Parkes Southern SERENDIPParkes Southern SERENDIP ALFA Sky Survey (300 MHz x 7 beams)ALFA Sky Survey (300 MHz x 7 beams) SETI Italia (Bologna)SETI Italia (Bologna) Astronomy Applications GALFA Spectrometer – Arecibo Multibeam Hydrogen SurveyGALFA Spectrometer – Arecibo Multibeam Hydrogen Survey Astronomy Signal Processor – ASP – Don Backer, Ingrid Stairs, (pulsars)Astronomy Signal Processor – ASP – Don Backer, Ingrid Stairs, (pulsars) ATA4 Correlator F EngineATA4 Correlator F Engine Reionization Experiments (Backer (UCB), Chippendale/Ekers (ATNF))Reionization Experiments (Backer (UCB), Chippendale/Ekers (ATNF)) Antenna Holography, ATNF, ChinaAntenna Holography, ATNF, China GMRT correlatorGMRT correlator

SERENDIP V Polyphase Filter Bank Server w/ EDT card GbE Switch PC Server w/ EDT card Server w/ EDT card Server w/ EDT card PC GbE Switch PC GbE Switch PC GbE Switch PC 100 MHz Pol. 1 Pol. 2 Astronomy Signal Processor – Don Backer

GALFA Spectrometer

sin cos LPF 100 MHz -50 to +50 MHz sin cos LPF 100 MHz -50 to +50 MHz Quadrature Downconverter Board IF Pol. 1 IF Pol. 2 Biplex 256 pnt. PFB e^-i  t FIR LPF FIR LPF 12.5 Mhz Digital Decimate by 16 Decimate by 16 Biplex 8192 pnt. PFB     Stokes cPCI Backplan e to CPU Multipurpose Spectrometer Board

GALFA Lowpass Filter

Mars Orbiter mm Spectrometer

ASIC based spectrometer (mars) ASIC based spectrometer (mars) 2W/ADC + 2W/ASIC = 4 Watts2W/ADC + 2W/ASIC = 4 Watts Use UCB’s “Chip in a Day” softwareUse UCB’s “Chip in a Day” software (compiles FGPA code into ASIC) (compiles FGPA code into ASIC) Use rad hard libraries from LBL Use rad hard libraries from LBL

Moores Law – Instruments using FPGA’s: 2X per year (1,000,000 over 20 years)

Future Spectrometers

Caveats RiskyRisky Simulink new, buggy, not open sourceSimulink new, buggy, not open source (verilog, vhdl old) (verilog, vhdl old) just a bunch of clever students, We’ve built the easy instruments so far, (Not the hard ones), yet to demonstrate packetized Correlator and compute cluster

CASPER the Friendly... Group Helping Open-source Signal- processing Technology (GHOST?)Group Helping Open-source Signal- processing Technology (GHOST?) –Goal to help develop signal processing instrumenation and libraries for the community. –Open source hardware, gateware, and software. –Provide training and tutorials –Not so much delivering turn-key instruments.