1. SKA Central Signal Processor Architecture SKA CSP LMC Peer Review 11.April, 2016, Madrid Sonja Vrcic SKA CSP LMC Sub-element Lead

2. List of topics to address as a part of this review CSP Architecture CSP Control Context Detail CSP Monitor and Control Architecture Prototyping - Status Proposed Design SKA CSP Monitor and Control Peer Review, 11. April 20162

3. Presentation - Overview and Approach Requirements and functionality for CSP_Low and CSP_Mid are similar but different. 1) This presentation first addresses CSP_MID: o Functions o First level decomposition o Sub-elements – functions and design 2) Discuss similarities and differences between CSP_Low and CSP_Mid. 3SKA CSP Monitor and Control Peer Review, 11. April 2016

4. SKA1_MID Telescope Source: Phase1 Baseline Design SKA CSP Monitor and Control Peer Review, 11. April 20164

5. CSP_MID Functions Receive input data from receptors Receive dual polarisation wideband digitised data from 133 SKA1- Mid receptors and 64 MeerKAT receptors. Resample and convert the MeerKAT data to a form compatible with SKA1-Mid. Input sample streams from all 197 SKA1 Mid and MeerKAT antennas are resampled into the corresponding common SKA1 Mid sample-rate. All the resampled signal streams can be processed by identical signal processing modules, reducing design effort. SKA CSP Monitor and Control Peer Review, 11. April 20165

6. SKA1_MID and MeerKAT Observing Bands Band Frequency Range (GHz) Bandwidth to be delivered to the SDP (MHz) as visibilities Sample Word-Length input to Mid.CBF Common Sample Rate (GSPS) input to Mid.CBF SKA1 Band 10.35 – 1.057008-bit4 SKA1 Band 20.95 – 1.768108-bit4 SKA1 Band 31.65 – 3.0514008-bit3.2 SKA1 Band 42.80 – 5.1823804-bit (TBC)12 SKA1 Band 54.60 – 13.802 × 25004-bit2 × 6 MeerKat UHF-Band0.58 – 1.015435TBD MeerKat L-Band0.9 – 1.6772010-bitTBD MeerKat S-BandTBD 1.712 MeerKat X-BandTBD SKA CSP Monitor and Control Peer Review, 11. April 20166

7. Sub-arrays SKA1-Mid operates concurrently in imaging mode and non- imaging mode with concurrent operation of between 1 and 16 subarrays. Each subarray is programmable as a separate conceptual telescope in terms of antenna pointing, band selection and the setting of configurable imaging and non-imaging parameters. Apart from limited resources (hardware that process input from receptors, forms tied-array beams and the communication links) sub-array are filly independent. SKA CSP Monitor and Control Peer Review, 11. April 20167

8. CSP_MID Functions Support for sub-arrays Make provision for TM to: a)Sub-divide array in up to 16 sub-arrays (group the receptors in up to 16 sub-arrays); b)Independently select signal processing mode and parameters for each sub-array; c)independently start/operate/stop signal processing in each sub- array. d)Assign any receptor to any sub-array; e)Assign any number of receptors to any sub-array, including a configuration where all receptors are assigned to the same sub- array; f)Perform signal processing concurrently and independently in all sub-arrays. Constraint: A receptor cannot belong to more than one sub-array at a time. SKA CSP Monitor and Control Peer Review, 11. April 20168

9. CSP_MID Functions Correlate input data received from receptors, transmit visibilities Simultaneously correlate and integrate input data for all receptors that belong to the same sub-array. Simultaneously and independently correlate input data in up to 16 sub-arrays. Send the visibilities resulting from the correlation to the SDP Element. The purpose of the correlation is to measure the level of signal correlation between all antenna pairs at different frequencies across the observing band. SKA CSP Monitor and Control Peer Review, 11. April 20169

10. CSP_MID Functions Correlation (continued) o Includes coarse delay compensation, channelization in frequency, RFI flagging and excision, fine delay compensation, cross multiply and accumulate. o Correlation is performed for each channel and each pair of antennas, including auto-correlation (where input from a single antenna is correlated with itself). o The result of this operation is commonly called the cross power spectrum, or visibility set and is a representation of the measured signal power from the sky as a function of frequency. SKA CSP Monitor and Control Peer Review, 11. April 201610

11. CSP_MID Correlator Output Products (Visibilities) SKA CSP Monitor and Control Peer Review, 11. April 201611 Number of channels: 64,000 Number of products: Nsub ‐ arrays x Nbeams x Nchannels x Nbaselines x Npolarisations For a sub-array where number of antennas N=197: 19,503*64,000*4 = 4,992,768,000 Integration time (requirement): 0.14 – 1.4 seconds Total correlator output 64 * 100 Gbps links. SKA baseline design:  The CSP ‐ Mid shall send and the SDP ‐ Mid shall receive visibility data at a maximum rate of 2.97Tbps.

12. Concurrency with other observing modes For the purpose of calibration, visibilities must be produced in all non-idle sub-arrays, but the number of channels may be reduced when correlation is performed concurrently with other modes. For example: Concurrency with VLBI beamforming is supported, but if performed in the same sub-array results in a reduced number of spectral-channels. SKA CSP Monitor and Control Peer Review, 11. April 201612

13. CSP_MID Function Zoom Spectral Line Imaging Simultaneously correlate and integrate input data for all receptors that belong to the same sub-array for:  up to 4 independently tunable spectral ‘windows’ and  across full bandwidth. Zoom windows are independently tunable :  Bandwidth configurable in range 4, 8, 16, 32, 64, 128, 256 MHz.  Center frequency configurable with the step size of 1MHz (±10%). Number of spectral channels per zoom window 16K. To reduce the data volume to SDP: adjacent regular spectral channels are channel-integrated to 1 ± 0.1 MHz coarse channels. SKA CSP Monitor and Control Peer Review, 11. April 201613

14. CSP_Mid Functions Phase binning For a single pulsar per sub-array, segment individual pulse period into equi-spaced ‘phase-bins’ in time (i.e. the accumulation units), integrate and send to the SDP. Make provision for TM to specify the pulsar period and the number of phase bins acquired. Note: o In this mode, the wideband delay-corrected signal is channelized into channels of width that allow for the number of prescribed phase bins and bin (time) width to be correlated, integrated and sent to the SDP. o 10 µsec bin width is possible for a range of pulsars and number of phase bins. SKA CSP Monitor and Control Peer Review, 11. April 201614

15. CSP_MID Functions Pulsar Search Beamforming Simultaneously form a total of up to 1,500 tied-array beams for pulsar search in up to 16 sub-arrays. o Any number of up to 1,500 pulsar search beams can be formed in any sub-array. o Any set of receptors that belong to a sub-array can be used to form any beam within that sub-array. Make provision for user, via TM, to specify the set of receptors to be used per beam and/or per sub-array. The purpose of the beamformer function is to coherently combine the signals from the MeerKAT and SKA1-Mid receptors such that the combined gain of those antennas is directed at a specified point on the sky. SKA CSP Monitor and Control Peer Review, 11. April 201615

16. CSP_MID Functions Search for pulsars and/or transients, transmit output products Search for pulsars and produce a list of potential pulsar candidates (and associated parameters) simultaneously for up to 1,500 pulsar search beams in up to 16 sub-arrays. Search for single pulses and produce a list of candidates and associated parameter data simultaneously for up to 1,500 pulsar search beams in up to 16 sub-arrays. Send the list of pulsar candidates, fast transient candidates and associated parameter data to the SDP Element. SKA CSP Monitor and Control Peer Review, 11. April 201616

17. CSP_Mid Functions Pulsar Search (continued) PSS beams are formed as the sum of selected antennas within ±10 km of the sub-array centre, are used to search for pulsars and fast transient sources. Acceleration search and single pulse search can be performed simultaneously in up to 1500 Pulsar Search beams (spread over up to 16 sub-arrays), each covering 300 MHz. Placement of beams within the input bandwidth is configurable per sub- array. Beamformer provides 4096 channels across each 300MHz beam. Pulsar search engine captures data samples for the pre-defined time interval and performs search. Duration of the search time-interval is configurable, range: 180 – 1800 seconds. Maximum output data rate: o pulsar search data: 32.66 Gbps. o fast transient (single pulse) search data 15.79 Gbps. SKA CSP Monitor and Control Peer Review, 11. April 201617

18. CSP_Mid Functions Form Pulsar Timing Beams Simultaneously form a total of up to 16 tied-array beams for pulsar timing in up to 16 independent sub-arrays. Any number of up to 16 pulsar timing beams can be formed in any sub-array. Any set of receptors that belong to a sub-array can be used to form any beam within that sub-array. Make provision for user, via TM, to specify the set of receptors to be used in beamforming per beam and/or per sub-array. Form Pulsar Timing beams using full input bandwidth for Bands 1, 2, 3 and 4 and half of the input bandwidth (2.5 GHz) for Band 5. SKA CSP Monitor and Control Peer Review, 11. April 201618

19. CSP_Mid Functions Time Pulsars, Transmit IPPs Produce Integrated Pulse Profiles (IPPs) and associated parameters for a single pulsar per beam, simultaneously for up to 16 Pulsar Timing beams. Make provision for TM to specify ephemerides and pulsar phase/longitude predictors for a single pulsar per PST-beam. Send Integrated Pulse Profiles and associated parameters to the SDP Element. SKA CSP Monitor and Control Peer Review, 11. April 201619

20. CSP_Mid Functions Form VLBI beams, Transmit VLBI beam data Simultaneously form up to 4 VLBI beams in up to 4 sub-arrays. Any number of up to 4 VLBI beams can be formed in any sub- array. Any set of receptors that belong to a sub-array can be used to form any beam within the sub-array. Transmit the VLBI beam data in a format that meets the requirements of the VDIF Transport Protocol. Concurrency: When performed concurrently with the VLBI beamforming in the same sub-array, correlation produces reduced number of spectral channels. Pulsar Search and Pulsar Timing can be performed concurrently with the VLBI beamforming, but not in the same sub-array. SKA CSP Monitor and Control Peer Review, 11. April 201620

21. CSP_MID Functions ECP: Transient Buffer Store raw or delay-corrected samples. TBC: Store up to 10 sec (TBC) of data samples (for the highest input data rate). When a trigger is received, send the stored data samples for all receptors that belong to the same sub-array to the SDP. Include timestamps in the output data packets. The trigger is generated by SDP and received via TM. SKA CSP Monitor and Control Peer Review, 11. April 201621

22. CSP_MID - Concurrency Up to 16 independently operated sub-arrays. Allow TM to configure, start and stop signal processing in each sub-array independently. In the table below, Observing Modes marked with x in the same column can be executed concurrently within the same sub-array. For the purpose of calibration, visibilities must be produced in all non-idle modes, but number of channels may be reduced (TBD). SKA CSP Monitor and Control Peer Review, 11. April 201622

23. CSP_Mid First level of decomposition Sub-elements: Correlator and Beamformer (Mid.CBF) Pulsar Search Engine (Mid.PSS) Pulsar Timing Engine (Mid.PST) Local Monitor and Control (CSP_Mid.LMC) Sub-element Integration Infrastructure (Mid.SII) SKA CSP Monitor and Control Peer Review, 11. April 201623

24. CSP_MID Context SKA CSP Monitor and Control Peer Review, 11. April 201624 E.1 100Gb/s each E.2 40Gb/s each Interface to SDP 64 * 100Gb/s links

25. CSP_Mid Correlator and Beamformer - Mid.CBF Functions Receive input from receptors Correlate input data received from receptors Perform pulsar phase binning Form tied-array beams for Pulsar Search Form tied-array beams for Pulsar Timing Form tied-array beams for VLBI Transmit visibilities, PSS, PST and VLBI beam data Support sub-arrays Store a limited amount of the raw or delay corrected samples received from each antenna and send to the SDP when trigger is received. SKA CSP Monitor and Control Peer Review, 11. April 201625

26. Mid.CBF Context Diagram SKA CSP Monitor and Control Peer Review, 11. April 201626 Source: Mid.CBF DDD

27. Mid.CBF Functions and Output Products SKA CSP Monitor and Control Peer Review, 11. April 201627

28. MID.CBF SKA CSP Monitor and Control Peer Review, 11. April 201628 Design: cca 300 LRUs, mostly custom, some COTS (servers, switches) Source: Mid.CBF DDD

29. Sub-arraying The diagram is a simplified representation of the correlator resources in the presence of sub-arrays. Correlator ‘cells’ that receive input from the antennas which belong to different sub-arrays are not used. Correlator resources are allocated indirectly, by assigning antennas to sub- arrays. SKA CSP Monitor and Control Peer Review, 11. April 201629

30. CSP_Mid Pulsar Search Engine - Mid.PSS Functions Receive input data from the CSP_Mid.CBF Search for pulsars (acceleration search) Search for transients (single pulse search) Transmit pulsar candidates and single pulse data products Support sub-arrays SKA CSP Monitor and Control Peer Review, 11. April 201630

31. Mid.PSS Design The PSS Engine consists of a cluster of COTS computer nodes. Each node is housing compute accelerators - Graphical Processing Units (GPUs’)and/or Field Programmable Gate Arrays (FPGAs). Each PSS node is designed to carry out all the functionality of pulsar search signal processing for up to two tied array beams. Number of nodes: Mid=750 (Low=250) +12 for contingency. cca 100 network switches (two per rack + aggregation of the PSS output). Monitor and Control Server, COTS server, dual redundancy. Total: 40 racks (cabinets) of equipment. SKA CSP Monitor and Control Peer Review, 11. April 201631

32. CSP_Mid Pulsar Timing Engine - Mid.PST Functions Receive input data from the CSP_Mid.CBF Produce Integrated Pulse Profiles (IPPs) Transmit IPPs Support sub-arrays SKA CSP Monitor and Control Peer Review, 11. April 201632

33. Mid.PST Context Diagram (Mid.PST DDD) SKA CSP Monitor and Control Peer Review, 11. April 201633

34. Mid.PST Pulsar Timing Engine Mid.PST consists of GPU-CPU compute clusters composed of common-use COTS equipment. 2-rack, 18-compute-node cluster to handle the processing of up to 16 tied-array beams independently (one node handling the processing for one beam + 2 nodes for redundancy). Mid.CBF each node must be capable of handling up to 96 Gb/s of data. Low.CBF each node must be capable of handling up to 15 Gb/s of data. PST M&C Server implements interfaces with CSP_Mid.LMC The compute load is observation dependent and largely determined by the requested dispersion measure and time resolution to process to. The maximum output data rate from the PST cluster will be ~10 Gb/s. SKA CSP Monitor and Control Peer Review, 11. April 201634

35. Mid.PST Hardware Components SKA CSP Monitor and Control Peer Review, 11. April 201635

36. CSP_Mid Local Monitor and Control - CSP_Mid.LMC Functions Support sub-arrays Implement interface with TM. Monitor, control and co-ordinate CSP_Mid sub-elements and report on behalf of CSP to TM. CSP.LMC provides a level of abstraction which allows TM to monitor and control CSP as a single system. Design: o COTS server, dual redundancy, o Network switch, dual redundancy, o Custom developed software. o Implementation and general functionality is the same in CSP_Mid.LMC and CSP_Low.LMC. o The difference is in the details of implementation: different parameters supported in each telescope. SKA CSP Monitor and Control Peer Review, 11. April 201636

37. CSP_Mid Sub-element System Integration Infrastructure Mid.SII The cost of equipment, tools and human resources required for integration of the CSP_Mid sub-elements are assigned to CSP_Mid.SII. SKA CSP Monitor and Control Peer Review, 11. April 201637

38. SKA1_Low Telescope SKA CSP Monitor and Control Peer Review, 11. April 201638

39. CSP_Low Functions – receive input data Receive input from 512 stations Input bandwidth 300MHz (50-350 MHz) Input from each station consists of 384 channels. Up to 8 beams per station (total number of coarse channels fixed). SKA CSP Monitor and Control Peer Review, 11. April 201639

40. CSP_Low Functions – overview of the observing modes Correlate input from 512 stations Number of channels per baseline 64K Up to 8 beams per sub-array Minimum integration time (requirement): 0.9 seconds Up to 4 zoom windows + bandwidth across full input BW Pulsar binning Form up to 500 pulsar search beams Form up to 16 pulsar timing beams (same as MID) No support for VLBI beamforming No transient buffer (LFAA responsibility) SKA CSP Monitor and Control Peer Review, 11. April 201640

41. CSP_LOW Sub-elements SKA M&C Harmonization Workshop, 11. April 201641

42. CSP_Low Design Low.CBF and MID.CBF are being designed by two different groups: o LOW.CBF design by CISRO, ASTRON and NZA, custom, FPGA based solution, Parentie technology. o MID.CBF design led by NCR-Herzberg with participation from MDA, NZA, custom, GPGA based solution, see www.powermx.org. Same design for LOW and MID: o Pulsar Search Engine, o Pulsar Timing Engine and o LMC SKA CSP Monitor and Control Peer Review, 11. April 201642

43. LOW.CBF Context SKA CSP Monitor and Control Peer Review, 11. April 201643

44. Low.CBF Overview - Design and Data Flow SKA CSP Monitor and Control Peer Review, 11. April 201644 cca 88 LRUs

45. SKA1 CSP to SDP Visibilities Data Rate (source CSP to SDP ICD) SKA CSP Monitor and Control Peer Review, 11. April 201645

46. CSP_LOW Output Data Rates The CSP-Low shall send and the SDP-Low shall receive visibility data at a rate in the range 2.49 to 3.73Tbps TBC. 64-bit integer UTC timestamp epoch of the last sample that was integrated (i.e. whether flagged/not correlated, or not). 32 bits is 1 second count since UTC epoch; 32 bits is fraction of a 1 second, providing ~0.25 ns timestamp resolution. The CSP-Low shall send and the SDP-Low shall receive pulsar search data at a maximum rate of 10.89 Gbps. The CSP-Low shall send and the SDP-Low shall receive fast transient/single pulse search data at a maximum data rate of 5.303 Gbps The CSP-Low shall send and the SDP-Low shall receive pulsar timing data at a maximum rate of 1.07 Gbps (this data rate is for the payload data only and does not include overhead for the file format, protocol, etc.) SKA M&C Harmonization Workshop, 11. April 201646

47. SKA CSP Monitor and Control Peer Review, 11. April 201647

48. Telescope Monitor and Control - Main Functions SKA M&C Harmonization Workshop, 11. April 201648 Planning, scheduling and execution of the astronomical observations. Management of telescope hardware and software sub-systems in order to perform the observations. Management of the data required to support operators, maintainers, engineers and science users in achieving operational, maintenance and engineering goals (excluding management of the science data products).

49. CSP Local Monitor and Control - Functions SKA M&C Harmonization Workshop, 11. April 201649 Each sub-system is interfaced to TM through a Local Monitor and Control (LMC) component in a standardised way. LMCs are connected to TM via a two-way TC/IP based network provided by SaDT, whose throughput is designed to handle the necessary data flows. Telescope configuration, dynamic status, calibration data and environmental data is time-stamped and stored, providing the current and historic state of the system (see Telescope Model, Section 10.1.2). Operators are presented with screens and reports that include real-time ‘system health’ status, progress of observations, on-site safety and security. Operators are provided with the communication and control tools to respond to changes in circumstances (e.g. changes in schedule, failures, emergencies, targets of opportunity).

50. CSP LMC Functions SKA M&C Harmonization Workshop, 11. April 201650 Configure: Set-up the configuration of the subsystem in question to operate in the commanded mode as well as set up the configuration of any monitoring or special functions. Control: For a pre-defined time, activate and provide parameters for any control actions required by the sub-system. Monitor: Activate, interrogate and receive monitor data from the sub- system. Provide Diagnostics: Activate diagnostics actions needed for maintenance of system ‘health’ and fault identification. Store Data: Accumulate and store monitor data. Some monitor data will require only short term storage. It may also be necessary to keep data that is not normally kept as part of the system state, but may be interrogated for diagnostic reasons. Such data might be kept in a circular buffer that overwrites itself after a predetermined time.

51. List of topics to address as a part of this review CSP Architecture (completed) CSP Control Context Detail CSP Monitor and Control Architecture Prototyping - Status Proposed Design SKA CSP Monitor and Control Peer Review, 11. April 201651

52. Sonja Vrcic NRC-Herzberg Dominion Radio Astrophysical Observatory – DRAO Penticton, BC, Canada Questions ? Thank you !