Status of EDI Archiving March, 2015 Leiden, Netherlands

Plan EDI team Status of instrument EDI products Delivery status Action Items

EDI operation is being done at UNH. UNH also supply us a set of cdf files (5 products). MPS team assembles the data, checks consistency and produces the CAA products(9 products). Braunsweig provides fgmcal data which is essential to make the EGD product. EDI team - management: PI: Roy Torbert, UNH, USA team:Hans Vaith, Mark Chutter, UNH, USA archive:Mikhail Rashev, MPS, Germany support:Karl-Heinz Fornacon, Institut für Geophysik und extraterrestrische Physik, Braunsweig, Germany, and Dragos Constantinescu, Institute for Space Sciences, Bucharest, Romania,

C1 – since operates in ambient electron mode because of interference with Whisper Jan 2001 – Jan 2010 operates with 1 gun, 2 gun is off. C2 – since operates in ambient electron mode because of interference with Whisper C3 – since operates with 1 gun because of interference with Whisper C4 – not operational EDI status EDI Concept Each spacecraft has 2 gun-detector units (GDU). Two guns emit a bunch of 1 keV electrons in opposite directions at different angles. Some of these electrons can return to detector traveling via circular orbit, according to ExB. Example values: Electron velocity ~ km/s Tg ~ 200 mks E ~ 2 mV/m Ne - ?

EDI products Products and purpose PPElectric Drift Velocity and Electric Filed, reduced-chi-squared value MSFraw data - headers and data packets from the various raw data files QSTATQuality Statistics for Electron Drift Velocity Measurements MPElectron Drift Velocity and Electric Field, Best Resolution PPPlusElectric Field, Drift Velocity, status, errors, drift step, „winners + loosers“ SPINElectric Drift Velocity and Electric Filed, spin resolution – major CAA product EGDElectron Gyrotime SUMMPLOTEDI Summary Plot AEAmbient Electron Counts AEDCCorrected Ambient Electron Counts AECAmbient Electron Correction Files CLISTCaveats CRFCode Repetition Frequency QZCBackground Electron Counts

product MSF, QSTAT DDDDDDDDDDDDDD CRFDDDDDDDDDDDDDD CLIST, EGDDDDDDDDDDDDDDP PP, MP, PPPlus, SUMMPLOT DDDDDDDDDDDDD AE, AEC, AE(cef), AEDC DDDDDDDDDDDDD SPINDDDDDDDDDDDDD QZCDDDDDDDDDDDDD D- delivered P- partially delivered R- reprocessing AE and AEC are being delivered by the US CoIs and (depend/do not) on NASA financing. AE/AEDC in CEF-format are produced at MPS after the delivery of AE and AEC products for the respective year. Current status of EDI data delivery

Data pipeline The data pipeline is divided between UNH and MPS. The origin files for data processing are msf files produced at the MPS. UNH uses these files to produce the first set of science data (5 products). MPS produces the rest (9 products) and sends all 14 products to CAA. Minor changes regularly but not frequently happen at UNH servers: server name, disks, software params. There is no change in EDI production software at MPS. However we change location of EDI files. We now change processing from 2 old machines (gcdc1 and gcdc2) to 2 new ones (gcdc3 and gcdc4). Gcdc1.mps.mpg.de ( ) – 2007 not in service by end of 2015 Gcdc2.mps.mpg.de ( ) – 2007 not in service by end of 2015 Gcdc3.mps.mpg.de ( ) – 2012 in service Gcdc4.mps.mpg.de ( ) in service, GCDC database All the following production for EDI is to be done on gcdc3.

EDI database at MPS Motivation: Upon request to check certain products and dates we found that EDI archive at MPS is not complete. So we would like to collect and to structure all files to be able to work on files massively through all years. /dev/sdb3 /data ~ 2Tb -current size 14 folders for 14 EDI products SPINAEDC /dev/sdb4 /back-up ~4Tb back-up image, 14 flags Technical goal: create save and consistent working zone via controlled back-ups, tracking changes in files and logs. Write programs for massive tasks: - Check data continuity through whole mission - data gaps, find files and lines with backward going time, find fill values - Write a template file to scan directories, files, find variable, line, value – for massive editing of both cef and cdf - Programs should be written in bash, perl, c/c++ with integration possibility (in a pipe) if required.

Action items CC19-AI-01Open check that the headers are complete and correct. This concerns particularly various descriptions in different levels (experiment, instrument, dataset). A: CC20-AI-08Open to clarify the instrument status on C1-2: are the guns OFF only due to a Whisper interference or for another reason This should be stated in the documentation and in instrument header (caveat) A: it seems the most headers are ok. How to check headers (general EDI header for each spacecraft) in CSA ??? CC20-AI-21Open to tell the CAA team which of the JSOC datasets and documents can be useful for the instrument teams (think also the post-operations phase) EDI10-AI-04Open Plot instrument settings panel: (a) AE/WW, (b) AE operation cycle, see 3.2 in UG … – CAA EDI11-AI-01Open to change the instrument coverage display in downloading GUI: ideally one should … – CAA EDI12-AI-02Open to investigate why EGD file has gaps while PP/SPIN/MP have data. Examples: C May 2005, Apr 2007, Sep 2009 (but plenty of similar events occur throughout the mission) MP has 13 records on 27 May (covering spin periods), so expects EGD and QZC records from the same 13 spins. However, records have varying quality (0, 1 or 2). Does this matter for the production of EGD and QZC? A: shoud be investigated together with Hans Vaith from UNH. He is the main EDI software author. EDI12-AI-03Open to investigate why QZC has gaps while PP/SPIN/MP have data. Examples: C May 2006 and 31 Mar 2007 (but plenty of similar events occur throughout the mission) A: shoud be investigated together with Hans Vaith from UNH.

EDI12-AI-05Open to check why there is a gap in AE but AEDC has data. Example: C2 – May 2007 Note that version numbering is very strange. In future if massive reprocessing is done, one could adopt the same version number which is higher than any existing version number, e.g. version 50 or 60. After this, new file versions will be upgraded by 1. A: a new versioning will be done in case of massive reprocessing. This case (v47) is exceptional – happened once. EDI12-AI-06Open to remove FILLVAL record from those SPIN files that have no other records. Instead, produce empty files for such days. See e.g. many days for C3 in Jan-Mar 2010 A: to be done EDI13-AI-01Open investigate why EGD and QZC are empty at the same time although MP and PP have data, see e.g June 2002 (C1). It looks like a processing error. A: shoud be investigated together with Hans Vaith from UNH. EDI14-AI-01Open to check and update the links in CSA website and to remove unnecessary links --CAA EDI14-AI-02Open to create a (simple) EDI homepage at MPS A: done. link: EDI14-AI-03Open to update the experiment and instrument metadata; check in particular the personnel, instrument caveats A: to be done identical, to CC19-AI-01 EDI14-AI-04Open to create a complete EDI database at MPS. Investigate first if the CAA database of EDI files is valid or if UNH has possibly produced new versions of files which were not transferred to the CAA. A: we started. AEDC files were copied from CAA. We are investigating techniques for how to build the archive at MPS.

EDI14-AI-05*Open to investigate how to remove unnecessary FILLVAL records from PP. At the moment there are daily files that contains (only) hours of FILLVALs. Note that this is not a consistent feature, sometimes empty files are produced. The logic of PP production is not clear. A: FILLVALS can be removed using cdfedit or by writing a program where CDFLib function is used. The logic should be investigated together with Hans Vaith from UNH because he is the main author of the EDI software. EDI14-AI-06Open to deliver the missing PPPlus files (plenty of files are missing) A: to be searched at UNH server. The production can only be done at UNH. EDI14-AI-08Open to deliver the missing MP, e.g. C3- Jan08, Oct11, Mar12, Oct12, Oct13. Produce a full investigation of missing MP. After the generation of EDI database at MPS, produce daily MP files for earlier years. A: to be searched at UNH server. The production can only be done at UNH. EDI14-AI-09Open to investigate if the log file on EDI operations/anomalies at UNH could be delivered to MPS who could convert it into CEF and deliver it to the CAA as the EDI Caveats dataset A: EDI14-AI-10Open to explain the source of the CLIST dataset to investigate why sometimes only CLIST has data, see e.g. C3-Apr’02 while all other datasets are empty (see also EDI14-AI-9). Is CLIST based on the Final PIOR or on the executed commands (i.e. taken from command history, or EDI HK) ? Is this a result of an anomaly that caused a removal of all data? A: ? EDI14-AI-11Open to investigate why sometimes only CRF and CLIST have data, see e.g. C1-Feb’05, May’08 while the other datasets are empty. This is a common feature, so is this another instrument operations anomaly that caused the removal of all data except for CRF and CLIST (see also EDI14-AI-9/10) A: to be done

EDI14-AI-12Open to investigate why sometimes only short transient data intervals were produced including CLIST (for an interval of a few minutes). Is this an instrument operations anomaly or was the instrument commanded to operate only for a few minutes with some purpose? See e.g. C1-Sep’03, Mar’05. A: need help of UNH EDI14-AI-13*Open to perform an investigation on the PP/SP production as long intervals of PP/SP are missing, see e.g. C3-May’11, Jan’12. A: to be searched on the UNH server. The production can only be done at UNH. EDI14-AI-14Open to clarify the SPIN production. One PP parameter (chi_squared) is removed but is something else done as well? A: CP_EDI_SPIN is a copy of the PP data without chi-squared. EDI14-AI-15Open to investigate why PP and SPIN do not always have the same coverage e.g. C3-Aug06. Normally they have identical coverage and time stamps. A: to be investigated. EDI14-AI-16Open to investigate why sometimes PP and SPIN have no data while MP has, e.g. C2-Jul02 A: to be investigated. In general the MP files contain more records then PP, SPIN. PP Electric Field, Drift Velocity, status, errors Winners with good/caution quality Spin resolution MP Electric Field, Drift Velocity, status and errors, Winners with good/caution/bad quality 1-4 s resolution,

EDI14-AI-17Open to investigate why EGD is empty (often QZC is empty as well) while E-field datasets (MP/PP/SPIN) contain data, e.g., C1-Oct02, Jan03, Jun03, Dec07, Oct09; C2- Jul01, Aug01, Sep01, Oct02, Jun03; C3-Nov06, Sep11, Oct11. This is not possible especially as MP rarely uses FILLVALs A: shoud be investigated together with Hans Vaith from UNH because he is the main author of the EDI software. EGD is produced from MSF files at MPS independently of PP/MP/SPIN. PP/MP are produced from MSF files at UNH. EDI14-AI-18Open to investigate why QZC is empty while all other science and aux datasets contain data, e.g. C1-Jun05 A: to be investigated EDI14-AI-19Open to investigate why AE and AEDC do not have consistent coverage, C1-Aug13, C2- Jan10. A: to be investigated EDI14-AI-20Open to consider a better policy for the file version numbers. Currently the largest version number used is v47. One could adopt version 50 if a regeneration of EDI files is done, and afterwards the version number is incremented by one if a new regeneration is done. A: The versioning is done manually. A new version will be created in case of massive reprocessing. the same as EDI12-AI-05. EDI14-AI-21Open to consider the best way of handling the datasets that are not produced any more, e.g. EDI E-field datasets on C1-2 for year CAA EDI14-AI-22Open to generate inventory analyses for AE & AEDC and for SPIN and PP. In both cases check time stamps and Nr of records –CAA EDI14-AI-23*Open to investigate the 125 timing errors recognized in AEDC files. Clarify why AE does not have timing errors as these two datasets should have identical time stamps. A: 2 files for one day. One file has timing errors. The other has not. See next page.

C3_CP_EDI_AEDC__ _0000_2359_V01.cef T01:05:48.957Z, T01:05:49.019Z, T01:05:49.082Z, T01:05:49.144Z, T01:05:49.207Z, T01:05:49.269Z, T01:05:48.863Z, T01:05:48.926Z, T01:05:48.988Z, T01:05:49.051Z, C3_CP_EDI_AEDC__ _V07.cef T01:05:47.988Z, T01:05:48.051Z, T01:05:48.113Z, T01:05:48.176Z, T01:05:48.238Z, T01:05:48.301Z, T01:05:48.363Z, T01:05:48.426Z, T01:05:48.488Z, T01:05:48.551Z, T01:05:48.613Z, T01:05:48.676Z, T01:05:48.738Z, T01:05:48.801Z, T01:05:48.863Z, T01:05:48.926Z, T01:05:48.988Z, T01:05:49.051Z, T01:05:49.113Z, T01:05:49.176Z, T01:05:49.238Z, T01:05:49.301Z, T01:05:49.363Z, T01:05:49.426Z, T01:05:49.488Z, T01:05:49.551Z, T01:05:49.613Z, T01:05:49.676Z, T01:05:49.738Z, T01:05:49.801Z, T01:05:49.863Z, T01:05:49.926Z, T01:05:49.988Z, T01:05:50.051Z, C3_CP_EDI_AE_ _v27.cef T01:05:47.988Z, T01:05:48.051Z, T01:05:48.113Z, T01:05:48.176Z, T01:05:48.238Z, T01:05:48.301Z, T01:05:48.363Z, T01:05:48.426Z, T01:05:48.488Z, T01:05:48.551Z, T01:05:48.613Z, T01:05:48.676Z, T01:05:48.738Z, T01:05:48.801Z, T01:05:48.863Z, T01:05:48.926Z, T01:05:48.988Z, T01:05:49.051Z, T01:05:49.113Z, T01:05:49.176Z, T01:05:49.238Z, T01:05:49.301Z, T01:05:49.363Z, T01:05:49.426Z, T01:05:49.488Z, T01:05:49.551Z, T01:05:49.613Z, T01:05:49.676Z, T01:05:49.738Z, T01:05:49.801Z, T01:05:49.863Z, T01:05:49.926Z, T01:05:49.988Z, T01:05:50.051Z,

EDI14-AI-25Open to deliver the EDI data from the commissioning phase. It is not clear which data products were produced. A: to be searched on UNH server. Ask UNH team after the very first phase of the MMS mission is over. EDI14-AI-26Open to describe the seven bits of status [3] in the EDI documents, e.g. in ICD. The information is available in the CSDS user guide. A: done. EDI14-AI-27Open to determine a conversion factor for EDI electron fluxes that can be used to convert AEDC and QZC count rates into electron particle and energy fluxes. One can determine a conversion for each orbit if data are available. A: The first answer from UNH is that: a number for the geometric factor of the EDI detector is found in the EDI instrument paper by Goetz Paschmann which I have attached (SSR 1997, THE ELECTRON DRIFT INSTRUMENT FOR CLUSTER). The number is documented in Table III on page 260 (page 28 of the pdf). The optics state we are using in both ambient and electric field mode is state 7, so the documented geometric factor H is cm^2 sr eV/eV. It is important to note that this number is just an average. The geometric factor of the detector varies quite a bit with elevation (polar look angle, which is set mainly with the upper four electrodes for the optics (upper/lower injector/deflector). The ambient electron cal files try to capture the polar angle dependence of the geometric factor. Comment: This is a very important information and should be added to the EDI CR. Action: to be done. EDI14-AI-28Open to update the UG: (1) fix the figure that overlaps with text; (2) modify text according to the comments and remove the footnotes; (3) check that the key OR review board recommendations are implemented (see their list in the MoM) A: done

EDI14-AI-29Open to check from the UNH if the key ground calibration reports could be delivered to the MPS and CAA archive A: Checked, a list of other questions. The UNH team is busy with MMS mission. Ask UNH team again after the very first phase of the MMS mission is over. EDI14-AI-30Open to update dataset headers: (1) QUALITY should refer to STATUS (E-field and drift velocity parameters) while now it states QUALITY=3 which is not correct; (2) use parameter caveat to describe complex parameters and their possible values A: It is not clear what the QUALITY refers to and why it is constant ? Could it be the value from qstat file. Then quality in qstat has a value of 3 Each QSTAT file has an introductory comment line that starts with a semicolon (;). The comment line contains the parameters that have been used when running the "qstat" program. There are two parameters: - min_q : the minimum quality required for recorded beams in telemetry. There are four quality values : 0 = no beam 1 = return beam detected, no time-of-flight information available 2 = return beam detected, time-of-flight information is available, signal-to-noise ratio is below upper threshold level 3 = like 2, but signal-to-noise ratio is above upper threshold level EDI14-AI-31Open to update the ICD: (1) fix the linking errors; (2) use the correct naming for the datasets, e.g. PPPlus shows varying naming; (3) improve the presentation of the header information A: done. OR2-R-32Open We encourage further cross-calibration of the AED with PEACE measurements at the same energy. OR3-R-21Open The board feels that calibration of AED is still an open issue. The team is encouraged to participate in a PEACE-EDI cross-calibration activity and to finish the remaining calibration tasks.

OR3-R-21Open The board feels that calibration of AED is still an open issue. The team is encouraged to participate in a PEACE-EDI cross-calibration activity and to finish the remaining calibration tasks. OR8-R13Open The board recommends that instrument teams ensure that all ancillary datasets are fully described in their ICDs. (See also CAA-OR8-R16 to CIS.).

Conclusion EDI science operation is being done at UNH. MPS assembles the data and produce CAA products. EDI is operational on 3 spacecrafts, mostly in passive mode. Electric field is being measured on spacecraft3. Delivery status: full delivery through Several products have been delivered for the whole Action items: there are many missing data gaps and products through that is to be investigated. We work on making EDI archive at MPS, we write programs for massive editing of both the cef files and cdf files.

We can build a model of EDI using Simion software. In order to do that we need EDI drawings and operating voltages on electrodes. Software: This model will simulate flight of electrons from gun to electron detector (time-of-flight, velocity vector, trajectory). We can simulate geometric factor, beam modulation, electron current, output values of EDI. If we want to model to give a realistic electrical signal then there are software to build a model of electronic circuit. Available software tools allow to build a complete model of scientific instrument: simulate media, simulate detection, simulate formation of electrical signal, output digits.

TCAD presentation: