20th CAA/DAA Cross Calibration Meeting Status of HIA-TC1 Data Archival H. Rème, I. Dandouras, A. Barthe Göttingen - October 17, 2014

Summary Management Status Data filtering Delivered Data Delivery Schedule Conclusion Gottingen - 17 october 2014 HIA Experiment Status

Management No change: Henri Rème: PI Iannis Dandouras: Co-PI Alain Barthe: Archive engineer We have no funding for this task and Alain Barthe is working on the Double Star data delivery in addition of his Cluster work. Gottingen - 17 october 2014 HIA Experiment Status

Status During the last months priority was given to finish our DAA contribution (competition with CAA for Alain). During this period our main task was to cleanup the HIA data, in order to provide the best possible science data to the archive. We produced and delivered data for 2004 and 2005 (1D and 3D products) Moments and 2006-2007 data will be delivered soon, before the end of 2014. DAA contribution for HIA will be fully ended this year. Gottingen - 17 october 2014 HIA Experiment Status

Data filtering To filter 3D spectra, we compute a scalar value (average of 3D counts). Plotting this function, we could detect single spikes, and create an algorithm to filter corrupted data. We can delete corresponding records to produce cleaned 1D and 3D products. Gottingen - 17 october 2014 HIA Experiment Status

Data filtering (example #1) Gottingen - 17 october 2014 HIA Experiment Status

Data filtering (example #2) Gottingen - 17 october 2014 HIA Experiment Status

Data filtering The algorithm gives good improvements in most of the cases. It could be improved to filter out more than single spikes. In some cases, data are too corrupted to be filtered (see next slides). These intervals are tagged as "NOISY" in our caveats database. Gottingen - 17 october 2014 HIA Experiment Status

Filtering (specific case) Gottingen - 17 october 2014 HIA Experiment Status

Filtering (wrong case) Gottingen - 17 october 2014 HIA Experiment Status

Data processing We have reorganised our processing chain to take in account data filtering. We systematically produce 4 kinds of CEF files: From original CDs and reprocessed (G version) Full data spectra and filtered ones. We compare from original and reprocessed: number of orginal records number of filtered records number of remaining records. Gottingen - 17 october 2014 HIA Experiment Status

Raw data selection Gottingen - 17 october 2014 HIA Experiment Status   Original CD's Reprocessed CD's Delta DATE INITIAL FILTER REMAIN 2006-08-26 2 10228 41 10187 10226 10185 2005-03-23 2058 4 2054 9516 16 9500 7458 12 7446 2006-11-20 4200 55 4145 9879 127 9752 5679 72 5607 2006-08-24 4783 9 4774 10038 22 10016 5255 13 5242 2005-11-12 2474 7 2467 7741 118 7623 5267 111 5156 2006-12-15 6636 14 6622 10045 21 10024 3409 3402 2006-12-05 6779 6766 3266 8 3258 2004-04-17 6102 239 5863 9382 626 8756 3280 387 2893 … 2004-09-05 2032 165 1867 2015 175 1840 -17 10 -27 2004-02-09 1062 3 1059 32 1030 29 -29 2005-01-23 7723 400 7323 7697 419 7278 -26 19 -45 2006-12-17 10261 23 10238 10239 74 10165 -22 51 -73 2006-02-07 6879 6867 4826 4818 -2053 -4 -2049 2006-12-04 5294 152 5142 2084 2071 -3210 -139 -3071 2006-11-27 10341 235 10106 6878 85 6793 -3463 -150 -3313 Gottingen - 17 october 2014 HIA Experiment Status

Data processing We added a plugin in our visualisation software (cl) to plot all these data in a same view (original and reprocessed, full and filtered data) We can show improvements of data reprocessing and validate data filtering. We have to check manually to decide the best source to produce data for a given day. Gottingen - 17 october 2014 HIA Experiment Status

Example (recovering data) Gottingen - 17 october 2014 HIA Experiment Status

Example (loosing data) Gottingen - 17 october 2014 HIA Experiment Status

Example (recovering noisy data) ? Gottingen - 17 october 2014 HIA Experiment Status

Last step: updating caveats The last step is to check/complete our caveats. We have to check data plots, day per day: deleting obsolete caveats (e.g: data gaps recovered by reprocessing.) replacing "noisy" intervals that become "cleaned" checking accuracy of old caveats, adding new ones Heavy job but we have now a good caveats database, with good accuracy. Gives read added value and could be delivered. Gottingen - 17 october 2014 HIA Experiment Status

Caveats (various) eclipse Gottingen - 17 october 2014 HIA Experiment Status

Caveats (various) Gottingen - 17 october 2014 HIA Experiment Status

Delivered data We have redelivered data for 2004 and 2005 Full 3D distributions x 5 physical units Omni-directionnal fluxes A previous delivery of HIA onboard moments for 2004 was done in May 2013. These data have to be redelivered after applying data filtering. Gottingen - 17 october 2014 HIA Experiment Status

Delivery schedule 3D distribution data for 2006 have been processed: from both old and reprocessed CDs before and after applying filtering a last check of caveats is still necessary They should be delivered in the next days. Data for 2007 could be produced in two weeks. Gottingen - 17 october 2014 HIA Experiment Status

Delivery schedule Onboard moments: We can apply the same algorithm as for 3D distributions, but we would want to improve it, to be able to cleanup more than single spikes. Using an algorithm with sliding windows should improve data filtering. A couple of weeks are necessary to validate the algorithm and produce data. Gottingen - 17 october 2014 HIA Experiment Status

Conclusion Data filtering on 3D distribution gives good improvements. It would be possible to complete data delivery until end of mission rapidly. Some work still necessary to produce momenta. A new caveats dataset will be produced. We would like to complete HIA-TC1 delivery by the end of the year, before switching completely back to Cluster. Gottingen - 17 october 2014 HIA Experiment Status