19th CAA CC WorkshopFrascati - 02/04 April 2014 CIS data delivery status 19th Cross-calibration Workshop Frascati, Italy - 02/04 April 2014

19th CAA CC WorkshopFrascati - 02/04 April 2014 Outline Data delivery status New datasets : Quality indices Software Data delivery plan

19th CAA CC WorkshopFrascati - 02/04 April 2014 Data delivery status Product CODIF 3D DDDDDDDDDDDDD CODIF 1D DDDDDDDDDDDD CODIF ancillary DDDDDDDDDDDD CODIF PAD DDDDDDDDDDDD CODIF ground moments DDDDDDDDDDDD CODIF corrected He + density (only C4) DDDDDDD CODIF-RPA 3D DDDDDDDDDDDD HIA 3D DDDDDDDDDDDD HIA 1D DDDDDDDDDDDD HIA PAD DDDDDDDDDDDD HIA Onboard Moments RRRRRRRRRRRR Graphical products DDDDDDDDDDDD CIS Modes DDDDDDDDDDDD CODIF_CAVEATS DDDDDDDDDD** HIA_CAVEATS DDDDDDDDDD** HIA Quality indexes DDDDD D = delivered P = partial delivery R = data redelivery

19th CAA CC WorkshopFrascati - 02/04 April 2014 Data delivery status 2013 CODIF data (C4 only) –CODIF 3D datasets have been processed with current calibration values, and delivered some days ago. –Remaining CODIF datasets will be delivered in the next weeks –Datasets depending on FGM data (PAD's) can be delivered until end of June HIA data (C1) –We have received recently a new set of calibration files, covering 2012/11/11 up to 2013/11/01 (cf. Iannis presentation) –Data will be processed and delivered in the next weeks

19th CAA CC WorkshopFrascati - 02/04 April 2014 Data delivery status CODIF and HIA caveats –These datasets are actually delivered on a yearly basis –Last delivery was rejected during CAA ingestion process –Each caveat covers a timerange that can exceed bounds of a given year, for example : T20:29:00Z/ T08:30:00Z, HIA, TLM, "Missing P2 and P4 products" –We would not want to truncate these events to make them enter in the bounds. –As these are not huge files, we can deliver a unique file (per instrument and per spacecraft) for the whole mission.

19th CAA CC WorkshopFrascati - 02/04 April 2014 New dataset : Quality indices We deliver 5 quality datasets : –one for HIA –one for each CODIF ion species (H+, O+, He+, He++) Each dataset contains : –time-tag (spin resolution from AUX SPIN TIME dataset) –4 quality flags (3D, PAD's, 1D, Moments) –caveat key(s) : string(s) indicating the origin of the problem

19th CAA CC WorkshopFrascati - 02/04 April 2014 Quality indices databases We create a quality events database : –a directory tree with a separate files per kind of event –different sub-directories per instrument (CODIF, HIA, FGM) –each file is a CSV database with the following variables spacecraft id time-range instrument (CIS = CODIF + HIA, CODIF, HIA, FGM) event key : RESET, ECLIPSE, PACC_DC, LOW_COUNTS… observation : additional information (string) –some files were manually completed during the whole mission (general caveats) –some are computed from specific algorithms

19th CAA CC WorkshopFrascati - 02/04 April 2014 Quality indices databases We are progressively adding new quality databases –writing specific algorithms –detect instrument caveats from housekeeping data (resets, CODIF post acc. discharges, HIA MCP discharges…) –from science data : CIS low counting statistics (density < threshold) FGM data for PAD's quality (ion gyroperiod > threshold) Each algorithm generates a new database : –With a dedicated event key –This key is associated with quality flags values (0,1,2,3) for each kind of flag (3D, PAD's, 1D, moments) –We mix all these databases to generate quality flags with a spin resolution

19th CAA CC WorkshopFrascati - 02/04 April 2014 Quality indices: key codes examples Key codeNote CODIFHIA 3DPad's1DMom3DPad's1DMom DISCRIWrong HIA discriminator levels 2222 MCPWrong HIA MCP levels 1111 MCPWrong CODIF MCP HV levels1111 MCP_DCHIA MCP discharge 1111 MCP_DCCODIF MCP discharge1111 PACC_DCCODIF Post acceleration discharge1111 TMInstrument mode mismatch CALIBCODIF calibration mode1111 CALIBHIA calibration mode 1111 ECLIPSEEclipse13 RESETInstrument reset NO_DATACIS Off or in mode 7 or LOW_COUNTSCODIF poor counting statistics2232 LOW_COUNTSHIA poor counting statistics 2232

19th CAA CC WorkshopFrascati - 02/04 April 2014 Quality indices examples (1/2) In the next slides, you will see examples of quality indexes related to instrument behavior : –ECLIPSE: specific algorithm and manual update for HIA, which has a different behavior from CODIF –Instrument RESET : computed from command history files and housekeeping data –HIA MCP discharge : computed from housekeeping data –CODIF PACC discharge : computed from housekeeping data

19th CAA CC WorkshopFrascati - 02/04 April 2014 Example: Eclipse

19th CAA CC WorkshopFrascati - 02/04 April 2014 Example: Instruments Reset

19th CAA CC WorkshopFrascati - 02/04 April 2014 Example: HIA MCP discharge

19th CAA CC WorkshopFrascati - 02/04 April 2014 Eg: CODIF PACC discharge

19th CAA CC WorkshopFrascati - 02/04 April 2014 Quality indices examples (2/2) A new set of quality indices computed from science data: –HIA low counting statistics density (computed with sliding average) < 0.05 cm -3 –FGM data used to qualify CIS PAD's B-field intensity < 8 nT Same kind of computation will be done for CODIF

19th CAA CC WorkshopFrascati - 02/04 April 2014 Example: HIA moments: low counting statistics

19th CAA CC WorkshopFrascati - 02/04 April 2014 Example: HIA PAD's

19th CAA CC WorkshopFrascati - 02/04 April 2014 Quality indices: summary HIA quality flags –Data for has been delivered for validation purpose –Still some work on metadata to link this dataset with other science datasets CODIF quality flags –First set of data were produced for H+ –Some work remains to refine flags for other ion species Improvements –We will add progressively new kind of quality events, in separate databases to refine quality flags –eg: Wrong CIS mode with respect to boundary location

19th CAA CC WorkshopFrascati - 02/04 April 2014 Software OR7-R04 "The board recommands that the CIS team provides a new release of the CESR CEFLIB software such that it compiles under Linux and copes with different line end character conventions." CEFLIB-v1.7.1 has been delivered to CAA (2013/09/04) : –Handling CEF files generated on Windows (CR+LF end of line terminators) –Handling a new datatype : BYTE used in PP files converted from CDF to CEF using QTRAN –Solving problem to build library on some recent MacOSX systems (gcc-4.2) –Renaming some files to avoid build conflicts on case insensitive file systems (MacOSX) –Adding a new interface with Python language : Users should install and use well-known numpy and matplotlib packages to manipulate and plot data.

19th CAA CC WorkshopFrascati - 02/04 April 2014 Software CEFLIB (2013/09/27) –New bug discovered recently : Nataliya BOURREL from CDPP team was unable to build the library on a Fedora19 computer. –This Linux release is embedding zlib package, which breaks compatibility with CEFLIB interface. –Changes were made to the zlib interface CEFLIB for Windows request –Andris VAIVADS, working with Yuri KHOTYAINTSEV team, asked for a version of CEFLIB working on Windows computer –Need an access to a Windows computer with available IDL and Matlab licences. CEF to NetCDF converter –The CDPP team asked for such converter, for the needs of AMDA software. –Fundings from CNES to develop this converter. –If anyone is interested by these tools, contact us.

19th CAA CC WorkshopFrascati - 02/04 April 2014 Software A dedicated CEFLIB WEB site has been opened – –Download library (including latest CEFLIB-v.1.7.2) –Installation instructions –User manual for the various interfaces (C, IDL, Matlab, Python) –Various topics : What's new, FAQ, Examples… Best way to keep up-to-date documentation

19th CAA CC WorkshopFrascati - 02/04 April 2014 Data delivery schedule Reprocessing data related to CIS calibration update: – HIA data: Recent calibration delivery – HIA data: Recent calibration delivery – CODIF data (O+ efficiency updates): waiting for calibraton files delivery Reprocessing data related to FGM reprocessing: –We have to write some tools to identify days impacted by this reprocessing