1. 20th CAA/DAA Cross Calibration Meeting
Status of HIA-TC1 Data Archival
H. Rème, I. Dandouras, A. Barthe
Göttingen - October 17, 2014

2. Summary Management Status Data filtering Delivered Data
Delivery Schedule
Conclusion
Gottingen - 17 october 2014
HIA Experiment Status

3. Management No change: Henri Rème: PI Iannis Dandouras: Co-PI
Alain Barthe: Archive engineer
We have no funding for this task and Alain Barthe is working on the Double Star data delivery in addition of his Cluster work.
Gottingen - 17 october 2014
HIA Experiment Status

4. Status
During the last months priority was given to finish our DAA contribution (competition with CAA for Alain).
During this period our main task was to cleanup the HIA data, in order to provide the best possible science data to the archive.
We produced and delivered data for 2004 and 2005 (1D and 3D products)
Moments and data will be delivered soon, before the end of DAA contribution for HIA will be fully ended this year.
Gottingen - 17 october 2014
HIA Experiment Status

5. Data filtering
To filter 3D spectra, we compute a scalar value (average of 3D counts).
Plotting this function, we could detect single spikes, and create an algorithm to filter corrupted data.
We can delete corresponding records to produce cleaned 1D and 3D products.
Gottingen - 17 october 2014
HIA Experiment Status

6. Data filtering (example #1)
Gottingen - 17 october 2014
HIA Experiment Status

7. Data filtering (example #2)
Gottingen - 17 october 2014
HIA Experiment Status

8. Data filtering
The algorithm gives good improvements in most of the cases.
It could be improved to filter out more than single spikes.
In some cases, data are too corrupted to be filtered (see next slides).
These intervals are tagged as "NOISY" in our caveats database.
Gottingen - 17 october 2014
HIA Experiment Status

9. Filtering (specific case)
Gottingen - 17 october 2014
HIA Experiment Status

10. Filtering (wrong case)
Gottingen - 17 october 2014
HIA Experiment Status

11. Data processing
We have reorganised our processing chain to take in account data filtering.
We systematically produce 4 kinds of CEF files:
From original CDs and reprocessed (G version)
Full data spectra and filtered ones.
We compare from original and reprocessed:
number of orginal records
number of filtered records
number of remaining records.
Gottingen - 17 october 2014
HIA Experiment Status

12. Raw data selection Gottingen - 17 october 2014 HIA Experiment Status
Original CD's
Reprocessed CD's
Delta
DATE
INITIAL
FILTER
REMAIN 2
10228 41
10187
10226
10185
2058 4
2054
9516 16
9500
7458 12
7446
4200 55
4145
9879
127
9752
5679 72
5607
4783 9
4774
10038 22
10016
5255 13
5242
2474 7
2467
7741
118
7623
5267
111
5156
6636 14
6622
10045 21
10024
3409
3402
6779
6766
3266 8
3258
6102
239
5863
9382
626
8756
3280
387
2893 …
2032
165
1867
2015
175
1840
-17 10
-27
1062 3
1059 32
1030 29
-29
7723
400
7323
7697
419
7278
-26 19
-45
10261 23
10238
10239 74
10165
-22 51
-73
6879
6867
4826
4818
-2053 -4
-2049
5294
152
5142
2084
2071
-3210
-139
-3071
10341
235
10106
6878 85
6793
-3463
-150
-3313
Gottingen - 17 october 2014
HIA Experiment Status

13. Data processing
We added a plugin in our visualisation software (cl) to plot all these data in a same view (original and reprocessed, full and filtered data)
We can show improvements of data reprocessing and validate data filtering.
We have to check manually to decide the best source to produce data for a given day.
Gottingen - 17 october 2014
HIA Experiment Status

14. Example (recovering data)
Gottingen - 17 october 2014
HIA Experiment Status

15. Example (loosing data)
Gottingen - 17 october 2014
HIA Experiment Status

16. Example (recovering noisy data)?
Gottingen - 17 october 2014
HIA Experiment Status

17. Last step: updating caveats
The last step is to check/complete our caveats.
We have to check data plots, day per day:
deleting obsolete caveats (e.g: data gaps recovered by reprocessing.)
replacing "noisy" intervals that become "cleaned"
checking accuracy of old caveats, adding new ones
Heavy job but we have now a good caveats database, with good accuracy.
Gives read added value and could be delivered.
Gottingen - 17 october 2014
HIA Experiment Status

18. Caveats (various) eclipse Gottingen - 17 october 2014
HIA Experiment Status

19. Caveats (various)
Gottingen - 17 october 2014
HIA Experiment Status

20. Delivered data We have redelivered data for 2004 and 2005
Full 3D distributions x 5 physical units
Omni-directionnal fluxes
A previous delivery of HIA onboard moments for 2004 was done in May 2013.
These data have to be redelivered after applying data filtering.
Gottingen - 17 october 2014
HIA Experiment Status

21. Delivery schedule 3D distribution data for 2006 have been processed:
from both old and reprocessed CDs
before and after applying filtering
a last check of caveats is still necessary
They should be delivered in the next days.
Data for 2007 could be produced in two weeks.
Gottingen - 17 october 2014
HIA Experiment Status

22. Delivery schedule Onboard moments:
We can apply the same algorithm as for 3D distributions, but we would want to improve it, to be able to cleanup more than single spikes.
Using an algorithm with sliding windows should improve data filtering.
A couple of weeks are necessary to validate the algorithm and produce data.
Gottingen - 17 october 2014
HIA Experiment Status

23. Conclusion Data filtering on 3D distribution gives good improvements.
It would be possible to complete data delivery until end of mission rapidly.
Some work still necessary to produce momenta.
A new caveats dataset will be produced.
We would like to complete HIA-TC1 delivery by the end of the year, before switching completely back to Cluster.
Gottingen - 17 october 2014
HIA Experiment Status