15th CAA Cross-calibration workshop CIS archiving activities report University College of London 2012, April 17-18

Outline Data delivery status Instrument status Recently delivered data Missing deliveries Software & documentation Filtering data & quality flags 15th CAA CC WorkshopLondon – 2012, April 17-18

Data delivery status 15th CAA CC WorkshopLondon – 2012, April 17-18

Instruments status C1 –CODIF switched-off since –HIA operating normally (magnetospheric modes) C3 –CODIF and HIA switched-off since C4 –CODIF operating normally –HIA switched-off since 2001 We have still to produce HIA data on C1 and CODIF data on C4 for the remaining years 15th CAA CC WorkshopLondon – 2012, April 17-18

Recently delivered data CODIF RPA data –This mode is only selected on some periods: 87 for C1, 153 for C3 and 64 for C4 for 2001 until 2009 –We decided during the 14th CC workshop to deliver a unique file per spacecraft and per year, to simplify deliveries –Each spacecraft files are provided for the 4 ion species and in the usual 5 physical units All data for C1 and C3 have been delivered We have still to provide data for C4 after th CAA CC WorkshopLondon – 2012, April 17-18

Recently delivered data CODIF LS data for C1 and C3 –LS mode was very rarely used on these spacecraft: 18 time intervals for C1 and 47 for C3 during the whole mission –These data are not well calibrated, but it was decided to deliver to CAA for archival purposes. –As for CODIF RPA data, we have delivered a unique file per spacecraft for the while mission All data for C1 and C3 have been delivered, remaining data for C4 will be delivered later 15th CAA CC WorkshopLondon – 2012, April 17-18

Recently delivered data Data for 2010 –We have delivered last week 3D datasets for 2010, using preliminary calibration files (very slow evolution of efficiencies since 2009) –This delivery covers the instruments still operational: CODIF on C4, and HIA on C1 –The remaining datasets (RPA, 1D, PAD’s and moments) will be delivered in the next weeks –These data will have to be reprocessed after calibration updates Data for 2011 could be delivered if necessary 15th CAA CC WorkshopLondon – 2012, April 17-18

Missing deliveries CIS09-AI-3 : To provide gap files for LS side of CODIF, on C1 & C3 –These GAP files are no more needed, as we provide now a single file per spacecraft for the whole mission for CODIF LS data –Same problem for CODIF RPA data : a single file per year and per spacecraft is now provided CIS-PM10-AI7 : To provide empty files for datasets that are not provided anymore (e.g. Cluster1 CODIF datasets since ) –A software tool has been created to provide such empty files, and we have to provide: CODIF data after 2004/10/25 on C1, CODIF and HIA data after 2009/11/11 on C3 HIA data on C4 –We have to provide such files for all king of datasets, all ion species and all possible physical units Data will be delivered in the next weeks 15th CAA CC WorkshopLondon – 2012, April 17-18

Software & documentation Documentation –CIS User guide: V.2.0 delivered (V.2.1 in preparation) –CIS Calibration report: V.1.2 delivered –CIS CAA ICD: V delivered Software –CEFLIB v1.6 (no changes) –Some installation problems between IDL interface and ZLIB library (on CentOS 5.6 and 6.x, x86_64 architecture), –solved by manual installation of zlib version in place of native –Iterating with some CAA users to help them to investigate problems 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering data In order to improve quality, it was decided during last CIS PM to filter some CIS data, in the following cases : –Eclipses –Weekly resets –Incomplete products due to telemetry lost packets (cf. also presentation on quality indices) We spent last weeks to identify such intervals, manually check and add them to our caveats database It will be soon possible –to deliver an updated caveats dataset for each instrument –to filter out some of these intervals when reprocessing data, –to use these caveats to define some quality flags 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering eclipses During eclipses CODIF is interpolating sun-pulse, and 3D distributions should be less accurate HIA is not able to make such interpolation, and could not compute 3D distributions. Moreover, HIA needs some minutes after long eclipses to be synchronized back with the sun-pulse, and 3D distributions are energy-shifted. We have to filter out these data or to indicate that problem with caveats and quality flags. 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering eclipses 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering eclipses 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering instrument resets Since the beginning of the mission, we are performing systematically resets of our instruments, approximately once a week. Each instrument is reset by a specific procedure, which progressively decrease HV, performs a software reset, and progressively increase HV back to the nominal value. We made a software tool that searches for reset commands in command history files, and search HV thresholds in Housekeeping data to identify such intervals. These intervals, different for CODIF and HIA, will be added soon in our caveats database. We will use these caveats to filter out these intervals when reprocessing data 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering resets 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering uncomplete products Lost telemetry packets –CIS 3D data are composed of several telemetry products, and each one can be sent in one up to 3 telemetry packets. –Some products are splitted in packets by ion species, other by polar bins, others by azimuthal bins –In some case, due to compression problems, some of these packets are not transmitted, and by the way we loose a part of 3D distribution We can find such events in our data processing logfiles, but human interaction is needed to define crearly which time intervals and high level products are impacted. 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering uncomplete products 15th CAA CC WorkshopLondon – 2012, April 17-18

Filtering uncomplete products 15th CAA CC WorkshopLondon – 2012, April 17-18

Quality flags We want to provide a quality dataset per instrument, with a given time resolution (1 mn ?), and with separate quality flags for each kind of products (3D, 1D, PAD’s, moments) and for each ion species for CODIF We will start to use the previous caveats to filter out data when reprocessing or to define quality flags for well identified time intervals. We will also check (manually) all the other kind of caveats in our database, to see their impact on each kind of products and possibly define corresponding quality flags. We are also planning to implement a new algorithm to detect some “poor counting statistic” intervals. Cf. presentation on CIS quality indices 15th CAA CC WorkshopLondon – 2012, April 17-18