Lessons from CASPER Correlators at GMRT Sandeep C. Chaudhari Back-End Development Group GMRT

CASPER Back-End Development Team Sandeep C. Chaudhari Shelton Gnanraj Mekhala Muley Kaushal Buch Harshvardhan Reddy Sanjay Kudale Irappa Halagali Indukumar Bhonde Ajithkumar B. Yashwant Gupta

Upgrade GMRT Digital Back-End Specifications Number of stations : 32 Max instantaneous BW : 400 MHz Number of spectral channels : 4-8 K Number of input polarizations : 2 Full Stokes capability Dump times : 1 sec or better Coarse and fine delay tracking : (+/-128uS) Fringe rotation : up to 5Hz Walsh switching Subarray support Incoherent and phased array beams (for pulsar work)

CASPER Hardwares at GMRT SERENDIP-V First generation CASPER hardware iBOB/iADC A second generation CASPER hadware ROACH A third generation hardware SERENDIP-V :- Developed a 4 antenna single pol correlator. Implemented and tested coarse delay and fine delay for it’s functionality. Provided a Microblaze control of delay correction to the design over UART via cPCI back plane.

CASPER Hardwares at GMRT SERENDIP-V First generation CASPER hardware iBOB/iADC A second generation CASPER hadware ROACH A third generation hardware iBOB/iADC :- Developed a 300MHz Wide band pocket correlator for iBOB and tested with antenna signals. Proposed digital back-end for 15- meter dish at NCRA. Used as a packetizer in GPU correlator. (SHR/GSJ)

CASPER Hardwares at GMRT SERENDIP-V First generation CASPER hardware iBOB/iADC A second generation CASPER hadware ROACH A third generation hardware ROACH:- Ported iBOB based Pocket Correlator to ROACH Automatic delay and fringe update added. Tested the design successfully with short and long baselines. Packetized correlator design modified as per GMRT specifications.

CASPER Hardwares – iBOB/iADC Developed a Wideband pocket correlator with the help of CASPER h/w & toolflow :- Wide Bandwidth = 300MHz Wideband FFT Block = 1024 point real. Wideband Coarse delay block designed at GMRT added to the design with delay correction up to +/-128uS Wideband fine delay and fringe stop block designed at GMRT added to the design. Fractional Delay correction = 1/32 of clock cycle. (11.25 deg over 512 channels) Max Fringe rate compensation = 32Hz. Tested the design for it's delay functionality using GMRT antenna signals viz. C02-C06 (baseline ~200m) along with modified 4-antenna design for SERENDIP-V system by GMRT group.

First Fringes with iBOB & SERENDIP-V Baseline ~ 200m & Delay 2.73uS (c. 2008) iBOB – Wideband PoCo First Fringes SERENDIP-V – Correlator First Fringes Uncompensated Delay Cross Spectrum Uncompensated Delay Cross Spectrum Compensated Delay Cross Spectrum Compensated Delay Cross Spectrum

ROACH – A matured CASPER Hardware Advancements in ROACH :- A modular hardware with external PowerPC control of FPGA hardware and ethernet connectivity for remote communication that makes the life simpler. Ample on-chip FPGA resources (BRAM) and onboard memory to accommodate final GMRT upgrade requirements. A reliable,robust and stable hardware platform with remote hardware monitoring and debugging facility provided. No JTAG programming required. Hence simultaneous multiboard programming without extra hardware is possible. MSSGE Toolflow of ROACH board has matured over time.

ROACH based Pocket Correlator Int Dly Blocks Fract Dly & Fstop Blocks

ROACH based Pocket Correlator Pocket Correlator :- A handy instrument for quick testing of Broadband analog system at GMRT. iBOB based pocket correlator ported to ROACH hardware. Automatic delay and fringe update script developed and successfully tested with broadband antenna signals for shorter baselines viz (C04-C11). Cross Phase Poco Norm Cross Amp Cross Phase Cross Phase GSB GHB Norm Cross Amp Norm Cross Amp

ROACH based Pocket Correlator The design successfully tested with broadband antenna signals for long baselines viz (C06-E06) approx.(13.2km)……!!! GSB Poco Cross Phase Cross Phase Norm Cross Amp Norm Cross Amp

Packetized Correlator built with ROACHES Packetized Correlator for GMRT:- A collaborative efforts of GMRT and KAT-SKA,SA group, in which a 4 antenna dual polarization generic Packetized correlator is developed by KAT-SKA,SA group and we modified according to GMRT upgrade requirements. Adopted XAUI based design of packetized correlator over 10Gbe approach. Changes as per GMRT requirements :- Changed from 2-Antenna 2 pols to 1-Antenna two pols per F-engine. Delay depth modified from +/-10.25uS to +/- 64uS (expanding to meet longest baseline delay requirements i.e. +/- 128uS) Corr control package is made compatible with XAUI design. DEVICE UTILIZATION :- Number of DSP48Es 269 out of 640 42% Number of BRAM 80 out of 244 32% Number of SLICE 25035 out of 58880 42% It is possible to fit +/-128uS delay depth requirement of GMRT for longest baselines and may be even 4k point FFT.

Packetized Correlator using ROACHES Lab test procedure developed and extensively tested with noise source. Long term stability tests carried out. Delay Tests for it’s functionality– Integer, fractional delay tested independently & then in combination with an automatic delay incremental loop. Tested for extreme boundaries. Fringe block functionality tests :- Initial theta and fringe rate tested independently and then in combination with an automatic theta incremental loop. Systematic lab tests results with noise source can be seen here.

Design flow & Test Setup at GMRT Signal Generator (CLK) Pin = 0dBm Fs = 800MHz Ant Broadband s/g from Optical Fiber Noise Source(Pin typ -13 to -15dBm) BW = 200MHz From10MHz Ref Broadband Signal Conversion & Amplification Automatic Dly update prog using astro_cal Corr ctrl & data acq PC Data Tx over 10/100Mbps link Data Dump in GMRT tax analysis prog format FILE CorrCtrl over katcp Data Analysis using GMRT tax prog Locked to NTP Server ROACH-I F-Engine 4-station Correlator F-engine : 4 ROACHES X-engine : 4 ROACHES 2 X-Engines XAUI Link 10Gbe Switch 1 PPS sync signal

Lab/Antenna Test Setup at GMRT Analog Signal Conditioning Setup Packetized Corr Setup Common Test Setup for FPGA-CPU/GPU-Packetized, PoCo

Packetized Correlator using ROACHES - Long Term Stability Long term stability test results with noise source. T= 3 hrs Cross Phase Norm Cross Amp

Packetized Correlator using ROACHES Design then successfully tested with broadband antenna signals with short and medium long baselines viz. C06-C11 and C06-S02 respectively. Frequency Band = 1280MHz @BW = 120MHz Frequency Down Conversion done at Baseband frequency.

Packetized Correlator Results Packetized corr Time series Plot Short baseline tests : Ants : C06-C11 (baseline ~500m) Source : 3c286 Cross Phase Norm Cross Amp Packetized corr self spectrum C11 Ant Packetized corr CX spectrum C06 Ant Cross Phase Cross Amp

Packetized Correlator Results Long baseline tests : Ants : C06-S02 (baseline ~4.3km) Source : 3c286 GSB Time series Plot Packetized corr Time series Plot Cross Phase Cross Phase Norm Cross Amp Norm Cross Amp

iBOB as a DBE for 15-m dish Long term stability issues :- Cross Phase Long term stability issues :- Normalized cross amplitude fluctuation over time observed. Asynchronous communication between iBOB and data acquisition PC. Caused due to corruption of self/cross data while reading by PowerPC & writing by FPGA to shared BRAM. Long term stability issue solved :- Provided internal integration counter, that is polled by acquisition program to ensure proper time stamped data reading. Norm Cross Amp Cross Phase Norm Cross Amp

Further Scope Packetized Correlator design need to be expanded to 8, 16 and then 32-station correlator in step by step manner. Existing Delay compensation range of +/-64uS to be expanded to +/-128uS Spectral resolution to be made better than 1MHz. Repeatability tests need to be carried out.

Lessons GMRT group has successfully built CASPER instruments like simple spectrometers and correlator to more complex systems as per GMRT DBE requirements and successfully tested using GMRT antennas for short and long baselines. Gained enough experience in designing, debugging and testing of CASPER over a course time that can be used while building next generation DBE for GMRT. Still miles to go……. A Scope for learning and contributing to CASPER.

THANK YOU!