1. CIS Data Archival Status
Alain Barthe, Iannis Dandouras and the CIS Team
CLUSTER ACTIVE ARCHIVE:
9th Operations Review
ESAC, 4 June 2014

2. Ion Composition and Distribution Function Analyser
1. The CIS Experiment
CODIF (CIS-1)
Ion Composition and Distribution Function Analyser
3D ion distributions with mass-per-charge composition determination
~0 – 40 keV/e Energy Range
HIA (CIS-2)
Hot Ion Analyser
3D ion distributions with high angular resolution
5 eV/e – 32 keV/e Energy Range

3. 2. Management The CIS CAA team is mainly composed of :
Iannis Dandouras: CIS PI and CAA scientific correspondent
Henri Rème: CIS Co-PI
Alain Barthe: Software engineer (CAA contract) CAA software and data processing Since July 2009 in part-time employment for CAA (3 days/week)
Emmanuel Penou: Software engineer; Graphics visualization tool (cl)
Other contributions from CIS Co-Is and IRAP (ex-CESR) members :
Sylvain Brunato: Operations and webmaster of
Lynn Kistler: CIS-1 (CODIF) calibrations (UNH, USA)
Chris Mouikis: CIS-1 (CODIF) special products (UNH, USA)
Adrian Blagau: (Institute for Space Sciences, Bucharest) : CIS-2 (HIA) calibrations and cross-cal activities

4. 3. Status of software development
CIS-3D-PAD
Software to produce CODIF and HIA pitch-angle distributions.
No recent changes
Omni-directional fluxes
Software to produce CODIF and HIA omni-directional fluxes (1D) datasets

5. 3. Status of software development
CIS-3D-MOM
Software to compute on the ground the ion moments, by reading CIS CAA 3D datasets.
No recent changes.
CIS-HIA-MOM
Software to produce the HIA ion moments from onboard moments
A bug was identified last year by a CAA user when comparing this dataset with CIS PP from CSDS: the CAA moments were erroneously time-tagged at the beginning of each spin, rather than in the middle as described in metadata.
Software has been corrected and data have been reprocessed, from the beginning of the mission, and delivered (September 2013) to CAA.

6. 3. Status of software development
CEFLIB
Software library written in C language to efficiently read CEF files from the various Cluster experiments, archived in the CAA.
Software interfaces were added for IDL and MATLAB.
Latest release, CEFLIB-V1.7.2, has been delivered on September 2013, with the following improvements:
new software interface with Python language
handling CEF files generated on Windows system (CR+LF end-of-line separator)
handling BYTE data type variables
solving library generation problems on some MacOS systems
into account modifications of zlib library interface.

7. 3. Status of software development
CEFLIB (continuation)
A dedicated WEB site ( has been opened :
access to the documentation (installation and user manual, FAQ)
examples of use with IDL, Matlab and Python
software download
user feedback
CEFLIB for Windows:
Request has been done by CAA users to deliver a version of CEFLIB that works on Windows computers.
Software is under preparation, and it would be possible to deliver binaries, but we need access to a computer with IDL and Matlab licenses to test and validate it.

8. 3. Status of software development
CEF to NetCDF Converter
The CDPP team asked for such converter, for the needs of AMDA software (Automated Multi-Dataset Analysis).
AMDA uses NetCDF format for its internal database, and this converter should give the availability to import all possible CAA datasets, and use AMDA functionalities to compare CAA data with other sources (CDAWeb, …)
This converter has been developed with CNES funding.
It is actually being tested by the CDPP team.

9. 3. Status of software development
Data delivery improvement
A set of scripts has been developed to ensure CIS data delivery completeness:
Automated requests to the CAA database to obtain CAA inventory listings for all CIS datasets (~ 300)
Comparison with IRAP internal catalogues
Automated detection of delivery discrepancies
Allows automated redelivery of missing files to CAA.

10. 3. Status of software development: CIS Data Quality Indexes
Five new datasets: four for CODIF (separate for each ion species) and one for HIA.
Separate index values, for each instrument, for each of the four dataset families :
moments
3D distributions
PADs
omni-directional fluxes
When the value of the index is below a given threshold, the corresponding data will be flagged.
In addition to the CAA-defined quality index, a CIS-specific flag (or caveat_key) is given, referring to the type of caveat and indicating thus the origin of the problem (described in the CIS User Guide).

11. 3. Status of software development: CIS Data Quality Indexes
CIS Quality Indexes :
-1: Removable data (CIS specific, for internal use)
0 : Not applicable (used by CIS to flag no-data tacking periods)
1 : Major problems, check caveats
2 : Minor problems, check caveats
3 : Good data
4 : Excellent data, has received special treatment
Default CIS Data Quality Index: 3 unless overwritten by another value, by software or manually.
More than one flags might apply to a given data interval; the data quality index is then the minimum of the values relevant to each flag.

12. 3. Status of software development: CIS Data Quality Indexes
In order to provide quality flags for the CIS datasets, we are using a quality events database:
a directory tree with separate files for each kind of event, and different sub-directories per instrument (CODIF, HIA, FGM, EFW)
files are CSV text files containing the following fields :
spacecraft ID
time-range
instrument affected (CIS = CODIF + HIA, CODIF, HIA) or instrument source (FGM for very low magnetic field, EFW for high spacecraft charging)
caveat_key (RESET, ECLIPSE, PACC_DC, ... cf. CIS User Guide: section 5.6)
observation : additional information
During the whole mission, we have registered manually some specific events in a main CIS caveats database.

13. 3. Status of software development: CIS Data Quality Indexes
We are now writing and using specific algorithms to automatically identify new kinds of events:
instrument caveats, from housekeeping data (instrument resets, telemetry problems, CODIF post-acceleration high-voltage discharges, ...)
using science data:
CIS low counting statistics (density < 0.05 cm-3)
FGM data used to qualify CIS pitch-angle distributions (B-field intensity < given threshold)
EFW data (spacecraft potential > 10 V, affecting CIS density accuracy)
Each new software module produces a new database, specific to a given problem.
Another software is then merging all these databases, to produce quality flags:
for each instrument and ion species (HIA, CODIF H+, CODIF He+, ...)
for each kind of data product (3D ion distributions, pitch-angle distributions, 1D distributions, moments)
with a spin resolution (taken from CAA SPIN TIME dataset).

14. Limitations related to operation incidents
Data Caveat
Identification
CIS Flag
CODIF
Quality Index
HIA 3D
PAD 1D
Moments
Limitations related to operation incidents
Wrong HIA discriminator levels
Caveats Database
and/or CMDH, HSK
DISCRI 3 2
Wrong HIA MCP HV level
Caveats Database and/or CMDH, HSK
MCP 1
Wrong CODIF MCP HV level
Caveats Database and/or: CMDH, HSK
HIA MCP HV discharge
Caveats Database and/or HSK
MCP_DC
CODIF MCP HV discharge
Example (cf. UG for complete list)

15. Limitations related to sampling of ion distributions
Data Caveat
Identification
CIS Flag
CODIF
Quality Index
HIA 3D
PAD 1D
Moments
Limitations related to sampling of ion distributions
Poor CODIF counting statistics
CODIF density value threshold : ~0.01 cm-3
LOW_COUNTS 2 3
Poor HIA counting statistics
HIA density value threshold : ~0.05 cm-3
He++ contamination from H+
All He++ data
HE2 1
O+ contamination when high H+
CODIF H+ density > 5 cm-3 and HS O1
CODIF on sc3 deficient anodes
All CODIF sc3 data after 23 Feb 2003
DEF_ANODES
Example (cf. UG for complete list)

16. Example: Eclipse HIA CODIF HIA quality index CODIF quality index
Depth in the shadow

17. Example: HIA MCP HV-discharge
HIA quality index

18. Example: CODIF PACC HV-discharge
CODIF quality index

19. Example: HIA counting statistics
HIA quality index

20. CODIF & HIA quality indexes
Example: Meaningless data due to a spacecraft-instrument TM mode mismatch
CODIF & HIA quality indexes

21. 4a. CIS Instrument Status
No Change
Rumba
Salsa
Samba
Tango
* 1 : Not operational since 25 Oct. 2004
* 2 : Not operational since 11 Nov. 2009
* 3 : to be switched ON, if necessary
* 4 : Magnetospheric modes only, since April 2011; operations reduced to 1 hour per orbit since November 2012

22. 4b. CIS Experiment calibration / cross calibration
CIS-CODIF Status and Calibration
CODIF calibrations, and cross-calibrations with WHISPER, PEACE, WBD (and CIS-HIA) are prepared at UNH, USA, by Lynn Kistler, Chris Mouikis and Jichun Zhang.
H+ and O+ Anode Efficiencies for have been determined.
H+ absolute efficiencies using rate data have been checked through June these look good.
Pressure balance tests have been done through Oct Still need to do 2012 and beginning These also indicate no significant changes.
There are indications that O+ efficiencies have been decreasing more than was estimated. This will require a correction back to ~2006 .

23. CODIF C4 Normalised Total H+ Efficiencies
HS - side
LS - side
After pressure balance test correction

24. Oxygen Efficiencies L=6-7 Density Ratios
Analysis of the ion composition during the long solar minimum showed some strange results.
The solar minimum points were going much too low.
L=6-7 Density Ratios

25. Plotting vs. time showed that although F10
Plotting vs. time showed that although F10.7 was relatively constant, the O+/H+ ratio was dropping significantly.

26. Oxygen Efficiencies Decided to try changing our assumption.
Decided to try assuming that the O+ efficiency change tracks the H+ efficiency change. Note that that was approximately true in the early mission, when the O+ beams were used.
Revised efficiencies shown with dotted line.

27. Original Data
Revised Efficiencies

28. 4b. CIS Experiment calibration / cross calibration
CIS-HIA Status and Calibration
HIA calibrations, and cross-calibrations with WHISPER, since 2009 are prepared at the Institute for Space Sciences (Bucharest) by Adrian Blagau.
Calibrations are now updated up to
For the more recent data (after 2012) the results show very stable efficiencies for the operating HIA sensor (C1, HS - side).
Update for the data, to refine the previously delivered calibration values for this period:
LS-side recalibration shows a correction of 1.87 % for sc1 and % for sc3, to be performed during data reprocessing.
HS-side recalibration shows a correction of 5.1 % for sc1 and 1.9 % for sc3, to be performed during data reprocessing.

29. HIA - WHISPER Cross-Calibrations Example: March 2013

30. 2012-2013: Very stable efficiencies
HIA Calibrations
(wrt 2011 efficiencies)
HIA Calibrations
(wrt 2011 efficiencies)
0.950
0.874
2012
2013
2011
2012
: Very stable efficiencies
(1 hour operation per orbit since Oct. 2012)

31. 5. Status of the dataset delivery
Product 01 02 03 04 05 06 07 08 09 10 11 12 13
CODIF 3D D
CODIF 1D
CODIF ancillary
CODIF PAD
CODIF ground moments
CODIF corrected He+
density (only C4)
CODIF-RPA 3D
HIA 3D
HIA 1D
HIA PAD
HIA Onboard Moments R
Graphical products
CIS Modes
CODIF_CAVEATS
HIA_CAVEATS
CODIF_QUALITY
HIA_QUALITY
D = delivered
R = reprocessed and redelivered
Delivered since the 8th OR

32. 6. Data delivery plan Missing files Short term (June - July 2014)
All previously identified missing files have been delivered.
Short term (June - July 2014)
Mass production and delivery of the remaining CODIF and HIA datasets.
Preparation and delivery of the data quality indexes datasets for CODIF.
Processing and delivery of the remaining HIA data quality indexes datasets for C1 and C3: years 2006 to 2013.

33. 6. Data reprocessing plan
Updates of calibrations files will imply reprocessing of corresponding time periods.
A massive reprocessing of the first years of the mission is necessary also for technical reasons (separate CEF header files).
Some reprocessing of the pitch-angle distributions is also required, for the few intervals for which FGM provided updated calibrations.
We would prefer to solve first the remaining HIA onboard software time-tagging problems:
we found several years ago a bug in the onboard software that randomly generates one spin number shifting in the HIA telemetry data
an algorithm was written to correct erroneous spin numbers and recompute HIA time-tags
we found a few periods were this algorithm doesn't work correctly
we have to investigate more, and understand how to improve it.

34. 7. Documentation CIS User Guide: V 3.4 delivered
CIS Calibration Report: V 1.5 delivered
CIS - CAA ICD: V delivered
Further updates of these documents under preparation, following the suggestions made during the 15 May CIS-CAA progress meeting.
3 CIS papers (2 for CODIF + 1 for HIA) are in the special issue of “Geoscientific Instrumentation, Methods and Data Systems”
More than ~ 70 science papers published per year, based on the analysis of CIS data. Total > 900 papers

35. 8.1 Replies to the recommendations of the 7th OR
CAA-OR7-INST-R4: The board recommends that the CIS team provides a new release of the CESR_CEFLIB software that it compiles under Linux and copes with different line end character conventions.
Done. Latest release, CEFLIB-V1.7.2, has been delivered on September 2013, cf. section 3.6 of this report for details.
CAA-OR7-R02 : The board recommends that all teams ensure that they have provided detailed information to the CAA regarding the pointing of instrument sensors relative to the spacecraft reference frame (e.g. sun- sensor or spacecraft Y-axis).
Done. CIS User Guide includes schematics of the CODIF and HIA instrument mounting and fields of view, with respect to the spacecraft. Furthermore, it specifies that the CIS 3D ion distributions are given in the ISR2 pseudo-GSE reference frame and that solid angles, defined in the distributions, correspond to particle arrival directions (direction of travel of the particle).

36. 8.2 Replies to the recommendations of the 8th OR
CAA-OR8-R13: The board recommends that instrument teams ensure that all ancillary datasets are fully described in their ICDs.
and
CAA-OR8-R16: The board recommends that all CIS datasets are fully described in the user guide and ICD, e.g. CP_CIS-CODIF_HS_SEL, CP_CIS- CODIF_LS_SEL, CP_CIS-CODIF_HS_RAT_IONS, CP_CIS- CODIF_LS_RAT_IONS, and GYRORAD quality flag.
CIS12-AI-11: To explain in the ICD what HS_TOF_EVENTS, HS_RATES and 64MQ are and what user can do with these data.
Done. A brief description of these ancillary datasets has now been added in the ICD, section In addition, document “Cluster CODIF Calibration Report. Part III: TOF Peak Analysis. Mass Threshold and Spillover Determination”, describing these data, has been delivered to the CAA. Concerning the quality flags, an Appendix has been included in the User Guide.

37. 8.2 Replies to the recommendations of the 8th OR
CAA-OR8-R14: All instrument teams should ensure that their user guides contain a detailed description and time-line of instrument operations and failures.
and
CAA-OR7-R03 : The board recommends that instrument teams provide information regarding special operations, ...., including their description and time of operation.
Done. Section 3.1 of the CIS User Guide contains a detailed description of all CIS operational modes and section 3.2 gives instrument operational status and noteworthy failures. The CIS-MODES data set contains a detailed time-line of instrument operations, and the CODIF_CAVEATS and HIA_CAVEATS datasets provide the detailed time-line of instrument operational incidents.
CAA-OR8-R15: The board recommends that all teams investigate data quality at mode changes and instrument switch-on or switch-off.
Done. The CIS-MODES data set contains a detailed time-line of all instrument mode changes and instrument switch-on or switch-offs.

38. 9. Status of Action Items
CC8-AI-22 : To include a quality factor to their moments data… (and related action items)
The first CIS data quality index files were produced and delivered this year to the CAA (HIA data quality indexes datasets for ). The HIA data quality datasets for the remaining years, and the CODIF data quality indexes datasets, will be produced and delivered soon.
CC18-AI-11 : To investigate how to implement the quality flags at spin resolution. This is important for the usage of quality flags that are meant for the filtering and identification of poor and high quality data points.
Done. The CAA/CSA Cx_AUX_SPIN_TIME dataset is now used as a time base for the CIS quality index datasets.
CC18-AI-13: To use a file listing flag in command line to get a listing of active files on the CAA.
Done. Automated comparison of CIS internal database and CAA inventory listing is now done after each new CIS delivery and ingestion in CAA.

39. 9. Status of Action Items
CC18-AI-8 : To investigate how to ensure that PAD’s and MOMENTS are regenerated if new FGM science files are redelivered. This can be done using command-line tool with ingested since flag
and
CC19-AI-04 : To reprocess the pitch-angle products for the reprocessed FGM dates.
We have started to identify the FGM source files used to produce each CIS PADs and HIA onboard moments CEF files (using CEF FILE_CAVEATS metadata). We have still to compare this data with CSA FGM dataset inventory to identify dates were data has to be reprocessed.

40. 9. Status of Action Items
CC19-AI-01: Check that the headers are complete and correct. This concerns particularly various descriptions in different levels (experiment, instrument, datasets).
CIS headers have been completed, particularly any reference to cesr.fr has been updated to irap.omp.eu. Experiment, instrument and dataset metadata have been checked and upgraded.
However, we have still to update some metadata to add a link to the CIS quality datasets for each of our science datasets, using the DATASET_CAVEATS attribute. CIS headers will be redelivered soon.
CC19-AI-14 : To investigate large offsets in the CODIF-4 He+ velocity in 23 Mar 2002, UT (see e.g. slide 11 in CC19-Annex-28). Is this a processing error? If not, is this kind of case flagged in quality?
Open.

41. 9. Status of Action Items
CIS08-AI-09 : To provide the estimate how the HIA background noise is changing in time. Such estimates would be highly useful to the user ....
and
CIS08-AI-10 (same, but for CODIF).
Open.
CIS09-AI-06 : To check if the problematic on-board compression intervals can be flagged automatically. UG or CR will include a figure (include PEACE/WHISPER density as a reference) with explaining text of this case; the related old AI is CC6-AI-3.
This is related to missing data products telemetry packets. This information will be included in the quality indexes: a quality flag (PACKET) has been defined for such cases. User Guide includes a figure (6.3) showing an example of the effect, on CIS data, of missing some telemetry packets for a given measurement.

42. 9. Status of Action Items
CIS10-AI-07 : To create (single) empty files for datasets that are not provided anymore, e.g. for Cluster 1 CODIF datasets since 25 October
To be delivered soon.
CIS12-AI-05 : To investigate why HIA/LS datasets end suddenly (no gap files); sometimes this happens to all files, see e.g. HIA-1 in March 2001.
Problem of CEF File_time_span metadata : In the first files, delivered at the beginning of CAA, File_time_span indicated the real coverage of data for a given day (e.g: 01:14:23/16:10:35) It indicates now the full day (00:00:00/23:59:59), even if there are no data inside. Problem will be solved with planned massive reprocessing of CIS data.
CIS12-AI-07 : To retrieve FGM SPIN files for the gaps in PAD (FGM-1 for Nov 2002, Sep 2006, Nov 2007).
Should be solved by the automated procedure that will be used to reprocess CIS data after FGM updates.

43. 9. Status of Action Items
CIS12-AI-10: To check the days when one of the CODIF ion species moments is missing (CODIF-1: Mar 2001, Nov 2001, Sep 2003; CODIF-3: Jun 2001, Oct 2008, Feb 2009, Mar 2009; CODIF-4: Sep 2003.
and
CIS12-AI-12: To check why CODIF-4 PAD LS H+ exists while no files for the corresponding He+ and O+ datasets.
Telemetry problem identified. Very low ratio of received/expected telemetry products, in some intervals where instrument is in BM telemetry and CIS mode 13 or 14. Some automated process to identity such intervals should be possible, and the intervals will be added in our caveats database and data tagged with quality flags. Cf. also action CIS09-AI-06.

44. 9. Status of Action Items
CIS12-AI-13: To clarify the He+ density production policy; e.g. CODIF-4 moments and corrected He+ density seem to have inconsistent content occasionally, see e.g. inventory plots for June and December 2001.
Done. CAA inventory plots have been checked for December CODIF corrected He+ density dataset is mixing CODIF HS and CODIF LS data. The inventory is the sum of CODIF HS moment and LS moment, and this is why inventory plots seem different.
CIS12-AI-14: Some 3D ion products are missing, e.g., CODIF-4: Feb 2006, Mar 2009, Jul 2010.
Done. CAA inventory plots have been checked for CODIF C4 on Feb 2006, March 2009 and July There is a mix of CODIF HS and LS data, but they seem OK and complete.

45. 10. Problems and open issues
Massive reprocessing for the first years of the mission will have to be done for several reasons: heavy process.
HIA time-tagging problem: bug in the onboard software that randomly generates one spin number shifting in the HIA telemetry data (need to improve the algorithm developed for fixing it during ground data processing).