Cluster Active Archive Status of DWP Data Activities Simon Walker, Keith Yearby, Michael Balikhin Automatic Control and Systems Engineering, University of Sheffield

21st CrossCal March, 2015, Leiden, Netherlands Management Michael Balikhin DWP PI Keith Yearby TM DWP TCOR, WBD BM2 Double Star Simon Walker Archive Scientist CORR, LOG, PIOR ICD, UM, CR Documentation for the new contract has been submitted to ESTEC

21st CrossCal March, 2015, Leiden, Netherlands Current Status TCOR –Data for all 2014 has been delivered. –Data for Jan, Feb 2015 has been delivered. DWP_LOG UT_PIOR –Delivered to end February CORR_FX, CORR_ST –Delivered to end April –Waiting for new FGM and AUX files. FGM ingested to Apr 2014 and Sep 2014 AUX SP ingested to April 2014 and Aug-Oct2014

Software Development 21st CrossCal March, 2015, Leiden, Netherlands Development and maintenance of software tools (internal to the DWP group) is on going. The current state of the tools is sufficient for the routine production of the DWP data sets and caveat files. UT_PIOR software required modification due to updated format. Data affected has been redelivered.

21st CrossCal March, 2015, Leiden, Netherlands Header Files No changes

21st CrossCal March, 2015, Leiden, Netherlands Documentation ICD User Manual Calibration Report No changes Some updates have been received from Arnaud and will be incorporated soon

Caveats 21st CrossCal March, 2015, Leiden, Netherlands No new files created. Software modifications after switching source to CSA.

21st CrossCal March, 2015, Leiden, Netherlands Delivery Schedule Year CORR 1 CORR_PLOT DDDD3/153/16 TCOR DDDDD4/16 DWP_LOG DDDDD3/16 UT_PIOR DDDDD3/16 D – Data delivered, 1 Depends upon FGM 5VPS and SP AUX data sets FGM available to end April, Sep 2014 AUX available to end April, Aug-Nov 2014

Action Items 21st CrossCal March, 2015, Leiden, Netherlands CC20-AI-02 to reprocess the pitch-angle products for the reprocessed FGM dates (for list, see slide 8 of annex 6). Response All data periods mentioned in this action item have now been reprocessed and delivered. CC20-AI-16 to investigate if and how the developed inventory tool could be used for their data set monitoring (see Annex 12). Response DWP already downloads inventories of newly ingested FGM, AUX, and DWP data sets.

Action Items 21st CrossCal March, 2015, Leiden, Netherlands CC20-AI-21 To tell the CAA team which of the JSOC datasets and documents can be useful to the instrument teams. Response The DWP team already provide a reformatted PIOR file. Other files of interest are LTOF for ephemeris information Master Science Plan OR9-R18 The board recommends that the DWP instrument team checks Figure 1 of DWP-UG with the new Appendix E of PEACE-UG for consistency in coordinate systems (cross references may be done if required). Response (CC20) This figure was taken from a previous PEACE UG. It is consistent with figures shown at the previous CrossCal. When the new version is available a comparison will be made.

WBD Processing 1 21st CrossCal March, 2015, Leiden, Netherlands The BM2 archiving task to be performed by Sheffield includes: Processing all new BM2 data acquired from 2015 onwards Processing the earlier BM2 data from 2010 to For the latter we will use level one files created by Iowa. It is planned that the BM2 archives will be created using software provided by Iowa. This generates CDF files, which may be converted to CEF using a CAA provided tool. At Iowa the software was run on a mix of Sun and Linux computers. At Sheffield all archiving software will be run on a Mac. Work to date has been concerned with porting and installing the archiving software. Excellent support from Iowa for this implementation.

WBD Processing 2 21st CrossCal March, 2015, Leiden, Netherlands Current status of modules mk_bm_files_v3f (C) Converts the TED output files into WBD level one format. Installed and working with no issues. Ephemeris database (SQL) Data produced by Prague, mirrored in Sheffield. Used for spacecraft position information on the plots, and also in the WBD planning files. GIF plotters (C) Produces the 30 second and 10 minute uncalibrated spectrogram (L1). Installed and working with no issues.

WBD Processing 3 21st CrossCal March, 2015, Leiden, Netherlands Current status of modules SpecPlotter (IDL and C) Produces calibrated overview plots from the level one data. Installed and working with no issues, except for obtaining an IDL licence to get full functionality. FGM-PEACE (C) Generates the ‘rotation’ files used to determine the antenna angles relative to the magnetic field direction and GSE axes (the name reflects the fact that this software was first produced to supply FGM support data to PEACE – it does not use PEACE data). Ported to Mac with some bug fixes, now working fine.

WBD Processing 4 21st CrossCal March, 2015, Leiden, Netherlands Current status of modules Cdf archive burst (C) Generates CDF files from WBD level one files. Ported to Mac with some bug fixes. Problem processing mixed TDA8/BM2 level one files fixed. Problem with WBD frames spanning hour boundaries fixed. Several issues with observations spanning date boundaries are still under investigation. Delta t between samples sometimes incorrect in digital filter modes. Modified to generate compressed CDF files directly, as there is no need for uncompressed files in the BM2 processing pipeline.

WBD Processing 5 21st CrossCal March, 2015, Leiden, Netherlands Current status of modules Two panel plotter (IDL and C) Plots spectrograms from CDF and level one data side by side. Installed on Mac with no issues. Problem with plots spanning CDF file boundaries partially fixed (can now plot across file boundaries, but the WBD frames actually spanning the file boundary are not processed). Angle plotter (Java / Autoplot ) Installed on Mac - no issues with code, just some difficulties with configuring the installation. CDF to CEF (binary executable only) Presently not working on Mac OSX ‘Mountain Lion’. CAA are investigating. The fall back position is that we will deliver CDF files, and let the CAA convert them.

WBD Processing 6 21st CrossCal March, 2015, Leiden, Netherlands Other software notes Some further validation of the CDF files has been performed using the CDF tool cdfstats. This provides statistics of CDF files including maximum and minimum values and number of fill values of each variable. The latter can be exploited to return the total number of records in a file (surprisingly there seems to be no tool to do this directly), as in the WBD CDFs, one or other but not both, of the E or B field samples are fill values. This revealed that some CDFs were slightly shorter than they should have been, due to part of the WBD frames spanning hour boundaries being missing, and that the delta t between samples was sometimes incorrect in digital filter modes. Issues identified porting from Sun to Mac were usually due to latent bugs in the original code (eg. writing outside the bounds of an array) that by chance did not cause program failures on the Sun.

WBD Processing 7 21st CrossCal March, 2015, Leiden, Netherlands Delivery and reporting The WBD BM2 datasets will remain WBD datasets under the control of the WBD PI. However it would be convenient in practice to for these datasets to be delivered via the DWP drop zone, and for WBD archiving activities performed by the DWP team at Sheffield to be reported in the DWP report. We assume that the Prague team will submit a separate report covering their archiving activities (for the WBD TDA8 mode data).