1. FGM Report: Extended Mode Processing 22 nd Cross Calibration Workshop Chris Carr, Patrick Brown, Leah-Nani Alconcel, Tim Oddy, Peter Fox, Cary Colgan (summer student 2015) Imperial College London 18 November 2015

2. FGM Team Chris Carr – PI Patrick Brown – Instrument Manager Tim Oddy – Instrument Operations Engineer, Systems Administration Leah-Nani Alconcel – Archive Engineer Peter Fox – Software Development, Calibration 222 nd Cross Calibration workshop November 18 th -19 th 2015

3. Primary FGM data delivered to CSA: Jan 2001 – April 2015 –Magnetic field data: Full resolution, 5 vectors/second, spin averaged. –Supporting files: GAPF (data gaps in processed data); VALF (data gaps introduced at validation stage – small number, generally only affects a few vectors at a time). Supporting FGM files delivered to CSA: Jan 2001 – April 2015 –CAVF: caveat files provided only when data quality is affected by external effects. Outlook –Routine calibration of remainder of 2015 will begin again after solar wind season starts in 2016 –GI embargoed data from Feb & Mar 2015 were delivered when the embargo period finished CAA Status of Data Submission 322 nd Cross Calibration workshop November 18 th -19 th 2015

4. Commissioning phase data Delivered first calibration of December 2000 data for C123 Redelivered data Data redelivered for a variety of reasons. A comprehensive list was provided in the FGM OR10 presentation for use by other instrument teams. Gap-filling Related to CC21-AI-05/06. Extended Mode processing FGM performed a trial processing of a single extended mode interval. We are now considering whether or not it is practical to do routine extended mode processing. Other recent FGM activities 422 nd Cross Calibration workshop November 18 th -19 th 2015

5. Delivered CAA data: December 2000 first calibration 5 SpacecraftOrbit Numbers DatesReason C12368-80Dec 2000First calibration of commissioning phase orbits from December 2000 for C123 22 nd Cross Calibration workshop November 18 th -19 th 2015 After CrossCal 21, it was determined that delivery of C2 data to the CAA was desirable despite signal issues, so it went in to join the C1 and C3 data delivered before CrossCal21. C4 data was not delivered because inboard sensor was used, and has not been calibrated.

6. Redelivered CAA data: December 2009 6 SpacecraftOrbit Numbers DatesReason C11450-1462Dec 2009Reprocessing and redelivery of C1 data so that HPA low mode data is reinstated. 22 nd Cross Calibration workshop November 18 th -19 th 2015 The FGM data taken when the HPA on C1 was in Low Mode were removed for the original delivery due to an offset shift. The offset shift has been corrected, as it has been routinely for C1 during calibration processing since January 2010. The December 2009 data were therefore reprocessed and the FGM data taken during HPA Low Mode have been reinstated.

7. Redelivered CAA data: July & August 2003 7 SpacecraftOrbit Numbers DatesReason C1234464-489Jul 2003 Aug 2003 Standard calibration found to be more realistic than additional inter-calibration 22 nd Cross Calibration workshop November 18 th -19 th 2015 QMUL flagged up a consistent problem in the July 2003 FGM data where the spin-axis calibration was suspect for C3. It was consistently offset from C124 by ~0.4 nT. Due to small s/c separations (~200 km), an offset of this magnitude would not be expected. The inter-calibration step that had been added to the standard calibration procedure was found to have introduced this offset in both July and August 2003 data. The standard calibration was therefore run on July and August 2003 and the FGM data redelivered to the CAA.

8. Redelivered CAA data: July & August 2003 822 nd Cross Calibration workshop November 18 th -19 th 2015 Orbit 473, 25 July 2003 interval, with intercalibration

9. Redelivered CAA data: July & August 2003 922 nd Cross Calibration workshop November 18 th -19 th 2015 Orbit 473, 25 July 2003 interval, without intercalibration

10. CAA inventory survey: Gap-filling 1022 nd Cross Calibration workshop November 18 th -19 th 2015 CAA provided a list of intervals of >30 mins in which the FGM SPIN files lacked data whilst PP files contained data for C1234 At CC21 it was determined that in at least some of these intervals, the PP data consisted of FILLVALs. Example was C1 on 21 September 2002. C1 had bad TM so FGM processing SW correctly removed this data. CAA had an action (CC21-AI-05) to provide new lists excluding intervals where PP data consisted of FILLVALs. This has not yet been completed. Nonetheless, FGM has performed an investigation on the C1 list.

11. CAA inventory survey: Gap-filling 1122 nd Cross Calibration workshop November 18 th -19 th 2015 12/32 intervals deliberately removed (yellow): mostly calibration sequences These cannot be reinstated

12. CAA inventory survey: Gap-filling 1222 nd Cross Calibration workshop November 18 th -19 th 2015 Of the remaining 20 intervals, 19 may have PP with FILLVALs 9 intervals actually have <30 mins of missing SPIN data where PP is present. Most are <15 mins. 7 intervals start or end on a day boundary – may be FILLVALs or a data pull issue (green) 1 interval has a mismatch between SPIN and 5VPS/FULL data (blue). This is the only interval that can definitely be reinstated

13. CAA inventory survey: Gap-filling 1322 nd Cross Calibration workshop November 18 th -19 th 2015 Total gap time where PP is present and SPIN is not (minutes) = 3396 Total gap time where PP is present and SPIN is not (hours) = 56.6 Approximate percentage of 13.25 years of data: <0.05% To put this into perspective, the gaps total about 1 orbit of data. Even if there are no FILLVALs, plugging these gaps increases FGM’s total mission coverage by considerably less than, say, adding FGMEXT data products would.

14. CAA inventory survey: Gap-filling 1422 nd Cross Calibration workshop November 18 th -19 th 2015 For the CAA’s regeneration of the >30 minute gap files, FGM requests the inclusion of the following conditions: Exclude intervals where PP has FILLVALs Exclude intervals where the VALF file has data – these have been manually removed for good reasons Exclude intervals which are 35 minutes in length AND in all four spacecraft – these are calibration sequences

15. Extended Mode Processing: Introduction 1522 nd Cross Calibration workshop November 18 th -19 th 2015 Extended Mode is: A patch of the flight software which allows FGM to buffer up to 23 hours of spin-averaged magnetic field vectors in instrument mass memory Allows FGM to record data even during periods when the spacecraft is not taking science telemetry – important as original Cluster science plan envisaged only 50% spacecraft data coverage After 15 years of operations, size of the Extended Mode dataset has become significant These data are NOT currently archived in the CAA

16. Extended Mode Processing: On-board data generation 1622 nd Cross Calibration workshop November 18 th -19 th 2015 Specific intervals of Extended Mode data have been processed in the past – required significant manual effort as Extended Mode data acquisition process differs from that of normal science data.

17. Extended Mode Processing: Time-Stamping 1722 nd Cross Calibration workshop November 18 th -19 th 2015 Normal science data are synchronised to the spacecraft Reset Pulse and are stored and time-stamped by the spacecraft. Extended Mode data are synchronised to the spacecraft Sun Reference Pulse and are stored by FGM – no time-stamping at acquisition time.

18. Extended Mode Processing: Time-Stamping 1822 nd Cross Calibration workshop November 18 th -19 th 2015 Gap (highlighted in yellow) between last normal science packet and first Extended Mode vector can vary. Accurate time-stamping requires careful examination of normal science and HK timelines. Automating this is a non-trivial task.

19. Extended Mode Processing: Calibration 1922 nd Cross Calibration workshop November 18 th -19 th 2015

20. Extended Mode Processing: Validation 2022 nd Cross Calibration workshop November 18 th -19 th 2015

21. Extended Mode Processing: Future Work 2122 nd Cross Calibration workshop November 18 th -19 th 2015 Continue working on extending FGM processing pipeline software to incorporate Extended Mode dataset. Identify and remedy sources of discrepancies in dataset calibration. Establish what limitations exist on the potential accuracy of dataset time- stamping. Work with Cluster Science Archive to make processed Extended Mode data products available to the Cluster science community.

22. FGM Progress Meeting Action Items 22 AI NumberAction ItemStatus FGM14-AI-07To update the UG, CR, and ICD according to the suggestions given during the PM Completed: New versions (i6.r1) have been delivered. FGM14-AI-04To create a new RDM database using the official Cluster RDM data; they can be retrieved online from the CAA. The teams should not use unofficial DDS data retrievals for the CAA data production. Submitted: FGM assumes that the 10- day data available on the DDS, which FGM uses for its database, is identical to that of the RDM, since the RDM is created at that point in time. Tim Oddy performed a comparison of DDS pulls and RDM data and found them to be identical. Possibly some data gaps early in the mission were caused by missing data in the DDS pulls. These can be filled through the CC21-AI- 05/06 investigation. Reprocessing the entire set of raw FGM data would require re-running the entire calibration pipeline (including visual inspection) on the resulting products, for which FGM does not have the resources. FGM does not believe that creating a new RDM database would add value to the FGM dataset. FGM12-AI-04to check why science data files contain gaps although PP contains data Submitted: This is redundant with CC21-AI-06 now. 22 nd Cross Calibration workshop November 18 th -19 th 2015

23. FGM Cross-Cal Action Items 23 AI NumberAction ItemStatus CC21-AI-19To comment on the new user interface of the data mining tool (see slide 1 of Annex 28) Submitted: No suggestions from FGM CC21-AI-10To check the metadata issues reported in Annex 17Submitted: FGM noticed that in the CALF files the contact is Patrick Brown. This should probably be changed to Leah-Nani Alconcel (Archive Engineer). CC21-AI-06To investigate the data gaps in SPIN while PP has data; intervals are listed in annex 15. Note that it has been recognized that in some of these intervals PP has FILLVAL (e.g. C1 on 21 Sep 2002); the investigation will be modified that only intervals with non-FILLVAL’s are reported. Open: FGM awaits the results of the modified investigation. Confirmed with TK that contents of Annex 15 still contain FILLVAL intervals. CC21-AI-05 on the CAA still needs to be completed. See earlier slides. CC21-AI-02To update the datasets that require FGM science datasets for the recently redelivered 45 orbits listed in annex 7 Submitted: This action would be recursive for FGM. CC20-AI-21To tell the CAA team which of the JSOC datasets and documents can be useful for the instrument teams (think also the post-operations phase) Submitted: None for FGM. CC19-AI-05To investigate why there are so many missing short NM intervals (see e.g. FGM inventory plots). Submitted: Once CC21-AI-06 is complete, only intervals <30 mins should remain. FGM will investigate if resources permit. 22 nd Cross Calibration workshop November 18 th -19 th 2015

24. 2422 nd Cross Calibration workshop November 18 th -19 th 2015 FGM Operations Review Action Items AI NumberAction ItemStatus CAA-OR8-R17The board recommends that the FGM team investigates the possibility of providing extended-mode FGM data to the CAA. Open: FGM has performed a trial processing of a small set of extended mode intervals. We are now evaluating whether or not it is practical to do routine extended mode processing. These data are transmitted during BM3 and contain the measurements when the spacecraft was not transmitting data. Time stamping of the data is a non-trivial problem as the data are given in instrument internal timing with no reference to the SCET time.