1. Correlators, Spectrometers, Beam Formers and VLBI using general purpose FPGA boards, tools & libraries (how to build eight radio astronomy instruments in two years) Dan Werthimer & Aaron Parsons University of California, Berkeley http://http://seti.berkeley.edu http://

2. Our research group is really 3 groups SETI (plus primordial black holes, HI mapping)SETI (plus primordial black holes, HI mapping) Public Participation Distributed ComputingPublic Participation Distributed Computing CASPER – Center for Astronomy Signal Processing and Electronics ResearchCASPER – Center for Astronomy Signal Processing and Electronics Research

3. SETI Group David Anderson, Bob Bankay, Court Cannick, David Anderson, Bob Bankay, Court Cannick, Jeff Cobb, Kevin Douglas, Josh Von Korff, Jeff Cobb, Kevin Douglas, Josh Von Korff, Eric Korpela, Matt Lebofsky, Dan Werthimer

4. UC Berkeley SETI Programs Name Time Scale Search Type SERENDIPseconds radio sky survey SETI@home mS - seconds radio sky survey Astropulse nS - mS radio sky survey SEVENDIPnS visible targetted SPOCK 1000 seconds visible targetted DYSON IR targetted

5. Public Participation Supercomputing Group David Anderson, Rom Walton, SETI Group David Anderson, Rom Walton, SETI Group aka Distributed Computingaka Distributed Computing aka “edge resource aggregation”)aka “edge resource aggregation”)

6. The SETI@home Client

7. SETI@home Statistics TOTAL RATE

8. BOINC: NSF Berkeley Open Infrastructure for Network ComputingBerkeley Open Infrastructure for Network Computing –General-purpose distributed computing framework. –Open source. –Will make distributed computing accessible to those who need it. (Starting from scratch is hard!)

9. Projects AstronomyAstronomy –SETI@home (Berkeley) –Astropulse (Berkeley) –Einstein@home: gravitational pulsar search (Caltech,…) –PlanetQuest (SETI Institute) –Stardust@home (Berkeley, Univ. Washinton,…) Earth scienceEarth science –Climateprediction.net (Oxford) Biology/MedicineBiology/Medicine –Folding@home, Predictor@home (Stanford, Scripts) –FightAIDSathome: virtual drug discovery PhysicsPhysics –LHC@home (Cern) OtherOther –Web indexing/search –Internet Resource mapping (UC Berkeley)

13. Where's the computing power? ● 2010: 1 billion Internet-connected PCs ● 55% privately owned ● If 100M participate: – 100 PetaFLOPs, 1 Exabyte (10^18) storage your computers academic business home PCs

14. CASPER: Center for Radio Astronomy Signal Processing and Electronics Research Henry Chen, Daniel Chapman, Pat Crescini, Christina DeJesus, Pierre Droz Kirsten Meder, Jeff Mock, Aaron Parsons, Andrew Siemion, Dan Werthimer Radio Astronomy Lab Don Backer, Paul Demorest, Matt Dexter, Carl Heiles, David McMahon, Mel Wright, Lynn Urry Berkeley Wireless Research Center Bob Broderson, Chen Chang, John Wawrzynek SETI Institute Dave Deboer

16. Casper Origins Casper Origins NSF proposal to build SETI spectrometer (2003)NSF proposal to build SETI spectrometer (2003) (added one paragraph: BTW, this can be used for other astronomy instrumentation, potential spin offs are ….) (added one paragraph: BTW, this can be used for other astronomy instrumentation, potential spin offs are ….) Reviewer’s comments (paraphased): ~”SETI is bullshit, SETI will never find anything, But these instruments are useful for the community, strongly recommend funding” But these instruments are useful for the community, strongly recommend funding”

17. CASPER Real-time Signal Processing Instrumentation (NSF ATI) Low NRE, shared by the communityLow NRE, shared by the community Rapid development (8 instruments / 2 years)Rapid development (8 instruments / 2 years) Open-source, collaborativeOpen-source, collaborative Reusable, platform-independent gatewareReusable, platform-independent gateware Modular, upgradeable hardwareModular, upgradeable hardware Industry standard communication protocolsIndustry standard communication protocols Low CostLow Cost

18. MOTIVATION ATA, SKA, Focal Plane Arrays, SETI, need >> PetaOp/sec Instruments take a long time to build, very high NRE

19. Allen Telescope Array 6.1-meter offset Gregorian (2.4-meter secondary) rim-supported, hydroformed dishes

23. ATA-42 Operational This Summer

24. The Radio Revolution

25. Inner core Station SKA Square Kilometer Array

27. The Problem with the Current Hardware Development Model Takes 5 yearsTakes 5 years Cost Dominated by NRE because of custom Boards, Backplanes, ProtocolsCost Dominated by NRE because of custom Boards, Backplanes, Protocols Antiquated by the time it’s released.Antiquated by the time it’s released.

28. Solution: Modular HardwareModular Hardware –Low number of board designs –Can be upgraded piecemeal or all together –Reusable –Standard signal processing model which is consistent between upgrades. is consistent between upgrades.

29. Solution: use FPGA’s 1 FPGA = 100 Pentium, 1/500 the power per op 3X improvement per year! Moores Law for FGPA’s

30. FPGA = Field Programmable Gate Array reconfigurable computing - 1 minute 100 times faster than CPU, 5 times less power integer arithmetic, not good at F. Point highly parallel (500 multipliers per chip) harder to program (mathlab simulink) tools to abstract the hardware away signal processing libraries

34. Compute Module Diagram

35. Platform-Independent, Parameterized Gateware What is Gateware?What is Gateware? –Design logic of FPGAs (between hardware and software) (between hardware and software) Need libraries for signal processing which don’t have to be rewritten every hardware generation.Need libraries for signal processing which don’t have to be rewritten every hardware generation. Matlab Simulink!Matlab Simulink!

37. Biplex Pipelined FFT Uses 1/6 the resources of the Xilinx module.Uses 1/6 the resources of the Xilinx module.

38. FFT controls (Verilog and Simulink Libraries) Transform lengthTransform length BandwidthBandwidth Complex or RealComplex or Real Number of PolarizationsNumber of Polarizations Input bit width and output bit widthInput bit width and output bit width twiddle coefficient bit widthtwiddle coefficient bit width Run-time programmable down-shiftingRun-time programmable down-shifting Decimate optionDecimate option

39. Filter Response: PFB vs. FFT

41. Additional PFB controls Filter overlapFilter overlap Width of filter coefficientsWidth of filter coefficients Window function for filter (hamming, hanning, etc.)Window function for filter (hamming, hanning, etc.) Import filter coefficients for custom filter performanceImport filter coefficients for custom filter performance Both FFT and PFB available as Verilog modules (no proprietary software, but not as portable between chips/architectures).

42. Digital Down-Converter Selectable # of FIR tapsSelectable # of FIR taps On-the-fly programmable mix frequencyOn-the-fly programmable mix frequency Selectable FIR coeffSelectable FIR coeff Agile sub-band selection.Agile sub-band selection.

43. X-Engine Correlation Architecture (Lynn Urry, Aaron Parsons)

44. X-Engine Architecture: applied to an arbitrary sized antenna array

45. Hardware and Software Libraries legend:

46. Applications

50. Global Interconnects Commercial Infiniband switch from Mellanox, Voltaire, etc.Commercial Infiniband switch from Mellanox, Voltaire, etc. –Packet switched, non- blocking –24 ~ 144 ports (4X) per chassis –Up to 10,000 ports in a system –200~1000 ns switch latency –400~1200 ns FPGA to FPGA latency –480Gbps ~ 2.88Tbps full duplex constant cross section bandwidth –<$400 per port

51. Beowulf Cluster Like General Purpose Architechture Dynamic Allocation of Resources, need not be FPGA based

52. Targeted Applications Moderate to high-bandwidth problemsModerate to high-bandwidth problems –For low bandwidths, just use CPUs Lower to mid-scale computationLower to mid-scale computation –For very large applications (SKA), may be more cost effective to design ASICs Rapid DevelopmentRapid Development

53. Applications Applications VLBI Mark 5B data recorder - Haystack – 500 MHzVLBI Mark 5B data recorder - Haystack – 500 MHz VLBA and Beamforming - CfA, Bob Wilson, Jonathan Weintroub VLBA and Beamforming - CfA, Bob Wilson, Jonathan Weintroub SETI – Arecibo (UCB)SETI – Arecibo (UCB) ATA (UCB, Seti Institute) ATA (UCB, Seti Institute) JPL/UCB/SI DSN 20 GHz, 2pol (Preston, Gulkis, Levin, Jones) JPL/UCB/SI DSN 20 GHz, 2pol (Preston, Gulkis, Levin, Jones) Correlators and Imagers:Correlators and Imagers: ATA (Mel Wright) ATA (Mel Wright) Reionization Experiment (Backer/NRAO) Reionization Experiment (Backer/NRAO) Carma Next Gen (Dave Hawkins, Caltech) Carma Next Gen (Dave Hawkins, Caltech) SKA demonstrator South Africa (Justin Jonas) SKA demonstrator South Africa (Justin Jonas) MWAR, LWA – MIT, NRL MWAR, LWA – MIT, NRL

54. 128 Million Channel SETI Spectrometer 200 MHz Bandwidth, 2 Hz resolution200 MHz Bandwidth, 2 Hz resolution

56. 1 GHz bandwidth “Pocket Spectrometer” Using ATMEL ADC’s at 2 Gsamples/secUsing ATMEL ADC’s at 2 Gsamples/sec Performing 4 real FFT’s in 1 (complex) biplex pipelined FFT module.Performing 4 real FFT’s in 1 (complex) biplex pipelined FFT module. 2048 channels2048 channels Uses just 1 ADC, 1 IBOB, and your laptop.Uses just 1 ADC, 1 IBOB, and your laptop.

57. Portable VLBI backend Interfaces to MARK 5B data recorderInterfaces to MARK 5B data recorder 500 MHz spectrum recorder.500 MHz spectrum recorder. (This makes 4 instruments in 1 year!)(This makes 4 instruments in 1 year!)

58. VLBI Mark 5B Front End 500 MHz BW, 32 channel filter bank VLBI Mark 5B Front End 500 MHz BW, 32 channel filter bank

62. Multi-Purpose FPGA-Based Spectrometer – Low Bandwidth Xilinx Virtex-II 6000 FPGA Xilinx Virtex-II 1000 FPGA 256 MB DRAM 200 Mhz ADC Compact PCI Backplane Software 200 Mhz ADC 200 Mhz ADC 200 Mhz ADC I I Q Q Pol. 1 Pol. 2 { { 200 Aux. I/O

63. SERENDIP V Spectrometer

64. SETI Applications JPL/UCB/SI DSN Sky Survey (20 GHz Bandwidth)JPL/UCB/SI DSN Sky Survey (20 GHz Bandwidth) Parkes Southern SERENDIPParkes Southern SERENDIP ALFA Sky Survey (300 MHz x 7 beams)ALFA Sky Survey (300 MHz x 7 beams) SETI Italia (Bologna)SETI Italia (Bologna) SETI@homeSETI@home Astronomy Applications GALFA Spectrometer – Arecibo Multibeam Hydrogen SurveyGALFA Spectrometer – Arecibo Multibeam Hydrogen Survey Astronomy Signal Processor – ASP – Don Backer, Ingrid Stairs, (pulsars)Astronomy Signal Processor – ASP – Don Backer, Ingrid Stairs, (pulsars) ATA4 Correlator F EngineATA4 Correlator F Engine Reionization Experiments (Backer (UCB), Chippendale/Ekers (ATNF))Reionization Experiments (Backer (UCB), Chippendale/Ekers (ATNF)) Antenna Holography, ATNF, ChinaAntenna Holography, ATNF, China GMRT correlatorGMRT correlator

65. SERENDIP V Polyphase Filter Bank Server w/ EDT card GbE Switch PC Server w/ EDT card Server w/ EDT card Server w/ EDT card PC GbE Switch PC GbE Switch PC GbE Switch PC 100 MHz Pol. 1 Pol. 2 Astronomy Signal Processor – Don Backer

67. GALFA Spectrometer

68. sin cos LPF 100 MHz -50 to +50 MHz sin cos LPF 100 MHz -50 to +50 MHz Quadrature Downconverter Board IF Pol. 1 IF Pol. 2 Biplex 256 pnt. PFB e^-i  t FIR LPF FIR LPF 12.5 Mhz Digital Decimate by 16 Decimate by 16 Biplex 8192 pnt. PFB     Stokes cPCI Backplan e to CPU Multipurpose Spectrometer Board

69. GALFA Lowpass Filter

71. Mars Orbiter mm Spectrometer

72. ASIC based spectrometer (mars) ASIC based spectrometer (mars) 2W/ADC + 2W/ASIC = 4 Watts2W/ADC + 2W/ASIC = 4 Watts Use UCB’s “Chip in a Day” softwareUse UCB’s “Chip in a Day” software (compiles FGPA code into ASIC) (compiles FGPA code into ASIC) Use rad hard libraries from LBL Use rad hard libraries from LBL

73. Moores Law – Instruments using FPGA’s: 2X per year (1,000,000 over 20 years)

74. Future Spectrometers

75. Caveats RiskyRisky Simulink new, buggy, not open sourceSimulink new, buggy, not open source (verilog, vhdl old) (verilog, vhdl old) just a bunch of clever students, We’ve built the easy instruments so far, (Not the hard ones), yet to demonstrate packetized Correlator and compute cluster

76. CASPER the Friendly... Group Helping Open-source Signal- processing Technology (GHOST?)Group Helping Open-source Signal- processing Technology (GHOST?) –Goal to help develop signal processing instrumenation and libraries for the community. –Open source hardware, gateware, and software. –Provide training and tutorials –Not so much delivering turn-key instruments.

77. http://seti.berkeley.edu/casper