Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT

Digital Packaging Processor Role of DPP Overall assumptions and priorities Interface Overview Tasks and algorithms Hardware/software Implementation DPP Interface Details

Data System Approach Fundamental Drivers Very limited software resources Non-trivial data rate and volume Automated analysis pipeline for efficient observing Must have science-useable system in place by September 2013 Data products to be readily useable by broader solar community –Data products with preset parameters –Data products with user-selected parameters –Tools and support for experienced users

Data System Approach Implications Monomode observing Calibrated data archived in application-specific databases Reliance on existing software packages –Miriad package for calibration & mapping –RHESSI SolarSoft package for user interface and data product display –RHESSI database model Err on side of over-rejection of data Limited initial support for ‘nice-to-have’ options Limited initial support for calibration refinements Limited support for non-solar applications

Data System Assumptions All information required for data analysis is written by the DPP to the Interim Data Base Engineering data acquisition, archiving and display is the responsibility of the ACC, and is “largely” decoupled from science data.

Nomenclature Data frame = Interval representing data from one correlator cycle (20 ms, ~4000 channels with 500 MHz range) Spectral frame = Data corresponding to a complete frequency-agile cycle (nominal 1 second, 10s to 100’s of ‘science channels, 18 GHz rang) Corresponds to a state frame. Scan: Observing interval within which target and frequency cycling pattern is unchanged

Role of Digital Packaging Processor To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base Real time, irreversible processing

DPP Interface Overview DPP State Frame ACC Correlator Start / End Scan Commands Scan-independent Calibration Parameters Scan Parameters Frame parameters Frame status report,, Correlations Interim Data Base Miriad format Internal RFI Database 1 s timing tick 0.02 s timing tick RFI results

DPP Task Timing Occasional – non operational –Accept, store and preprocess calibration parameters Scan initiation –Accept, store and preprocess scan-specific parameters Data frame (20 ms) –filter, and frequency-average correlator output Spectral frame (1 s) –Assemble, pre-calibrate, reformat and write data to Interim database TBD –Format results and write to RFI database

DPP – Stage 1 Processing Every data frame (20ms) Evaluate kurtosis data to identify RFI-affected subbands as a function of frequency only. Save RFI statistics Combine with pre-flagged subbands to generate a “destination vector” for each subband Apply complex gains at subband level ??? Average subband data into spectral channels Save 1 st 3 moments of averages ???

DPP Stage 2 Processing Every spectral frame (1s) –Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags –Apply time-independent complex gains if available –Apply baseline corrections –Apply non-linearity corrections –Correct for attenuator settings –Correct for spectral simultaneity Miriad format  this is no longer optional –Convert visibility, uv and analysis-relevant state-frame data to Miriad-compatible format –Write spectral frame to IDB –Report DPP status to state frame

DPP - Implementation Original concept was to follow FASR plan for a cluster-based DPP Estimate processing requirements for EOVSA DPP at ~100 MIPS = 1/60 of FASR requirements Implementation will be based on a single multi-core machine Software organization will be compatible with migration to a cluster if necessary

DPP Software Architecture ACCState Frame Correlator IDB Coordination Task I/O, data assembly, no processing per se Header Processing Stage 1 Processing Stage 2 Processing Pointers within shared memory Conventional, time-independent processing tasks C1 C2C3, C4C2 Cn = core within a quad core processor or nodes in a cluster DPP RFI database Parameter Processing C2

DPP Status Software architecture and tasks identified Detailed definition of interfaces is underway EOVSA to Miriad format conversion being tested with FST data –(Fortran 77 for Miriad compatibility) Next: 1. Complete definition of interfaces 2. Code Stage 1 tasks (GH) Evaluate timing requirements 3Code Coordination task (JM). 3.Detailed definition of processing algorithms 4.Code of Stage 2 tasks 5. Machine selection and purchase Development platform? Goal: Functional DPP to support prototype testing