Overview of the CAA Activities H. Laakso

Summary of Activities CAA development activities in 2013/4 have primarily focused on maintenance and optimization of the existing core system support for the CAA to CSA migration updating of visualization tools production of cross-calibration comparisons Development of new software for the Double Star datasets; EFW science datasets; velocity moments in inertial frame in both GSE and ISR2; bow shock crossings/solar wind intervals production of new auxiliary datasets, including ESOC datasets Interaction on deliveries of data from non-Cluster sources: Double Star - ongoing EU FP-7 ECLAT (European Cluster Assimilation Technology) - completed MAARBLE (Monitoring, Analyzing and Assessing Radiation Belt Loss and Energization) - ongoing

Status of File Transfer to CSA http://caa.estec.esa.int/caa/csa_stats.xml

Status of CAA Hardware

Management and Standards

Management

Cross-Calibration Workshops The organization of regular cross-calibration workshops, even at this stage in the mission, has been a vital factor in the successful implementation of the CAA and this is an important strategic lesson for any space science project in the future. Two cross-calibration workshops were held since the last review (October 2013 in Abingdon, and April 2014 in Frascati) An overview of these meetings is given in document CAA-EST-RP-0102 The minutes of the meetings with all presentations can be found at http://caa.estec.esa.int/caa/cross-cal.xml In addition the minutes of the meetings from these two workshops are attached to the review package as reference documents

Instrument Progress Meetings An annual progress meeting was held between CAA and each team during Nov 2013 - June 2014. A normal agenda of the progress meetings is 1. Welcome 2. Status of the instrument datasets in the CAA 3. Status of Action Items a. Status of action items from the Progress Meetings b. Status of action items from the Cross-Calibration Workshops c. Status of the OR Recommendations Status of data production and delivery Status of Calibration and cross-calibration activities Status of Double Star archiving 6. Status of Documentation (UG, CR, ICD) 7. AOB The minutes of the progress meetings are attached to the review package the minutes of the meetings with all presentations can be found at ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/ProgressMeetings/

Management of Action Items Due to a large number of action items from Instrument Progress Meetings and Cross-Calibration Workshops, the management of action items has been challenging. To help their management a new GUI was developed. The page can be accessed at http://caa.estec.esa.int/caa_action/action_list.xml

CAA Metadata Dictionary, v3.0 New version, v3.0, can be downloaded at CAA/Documentation Previous release, v2.3, was published on 5 March 2008 and is outdated New document name (CAA-MDD-0001) because MDD is now maintained by the CAA Style is a bit simplified & changed in order to ease its reading & browsing Key updates Several updates in enumerated values for the following keywords: MISSION, MISSION_AGENCY, EXPERIMENT, OBSERVATORY, INSTRUMENT_NAME for the Double Star mission New values for MEASUREMENT_TYPE, INSTRUMENT_TYPE, PROPERTY, TENSOR_FRAME Modified definition for QUALITY New coordinate systems: FAC, MFA new keyword: TARGET_SYSTEM Changes in appendices Appendix D deleted Enumerated lists are now given in Appendix B

Update of the CAA Metadata Dictionary, v3.0

Dataset Production, Delivery and Ingestion Status of Dataset Production, Delivery and Ingestion

Cluster Datasets – Delivery Status The baseline of the data delivery is to provide all 2013 data by the 9th Operations Review Various challenges exist such as there are either difficult calibration issues with aging instruments, or there are important instrument dependencies in the data production

Cluster datasets - Critical Datasets There are four critical datasets that are needed by many other teams for their data processing and calibration activities Realistically one cannot expect the data deliveries sooner than ~6 months after the delivery of these critical datasets. The delivery status of the critical datasets is : DWP time correction: up to Dec 2013 EFW spacecraft potential: up to Sep 2013 FGM science datasets: up to Feb 2014 WHISPER density: up to July 2013

New Datasets: Velocity Moments Created for PEACE and CIS Parallel and Perpendicular Velocity in inertial frame in both GSE and ISR2 (spin-averaged) DC Electric Field in inertial frame in both GSE and ISR2 (based on PerpVelocity) The magnetic field used in all calculations is FGM_5VPS from which the magnetic field is determined for the time interval of the given velocity moment The parameters in the new velocity datasets are: Data_Velocity_GSE_inert Data_Velocity_ISR2_inert Data_PerpVelocity_GSE_inert Data_PerpVelocity_ISR2_inert Data_ParVelocity_GSE_inert Data_ParVelocity_ISR2_inert Data_Efield_GSE_inert Data_Efield _ISR2_inert Hourly comparisons between EFW, EDI, PEACE and CIS created for DC E-fields and Drift Velocities Plots can be downloaded or viewed via QL Browsing Tool These variables can be compared with EDI and EFW

DC E-field at magnetopause

C1: Drift Velocity in the plasma sheet

C3: Drift Velocity in the plasma sheet

Detection of Boundary Crossings The work on the detection of the boundary crossings by Cluster is at its final stage Data processing and detection is done with C Plotting is done with matlab The first boundary to be determined is the Bow Shock Required step-like changes must happen in both Bt and HIA Vx within a few minutes if they available If only one of them is available, then two other steps are required (several parameters are investigated) Other boundaries: The following work will be the radiation belt crossings which is expected to happen quickly The magnetopause crossings will be the last one User recommendation: it is better to generate list of region intervals rather than the boundary crossings, e.g. list of intervals when the spacecraft is in the solar wind This is not trivial as the spacecraft can be off during a bow shock crossing

Example: Bow Shock Crossings Script is working quite well but human validation is needed in order to add new boundaries and remove wrong boundaries

EU/FP7 ECLAT datasets The EU FP7 ECLAT (European Cluster Assimilation Technology) project was completed in Feb 2014 ECLAT datasets are being transferred to the CSA

ECLAT Quicklook Plots ECLAT Quick-look Plots

EU/FP7 MAARBLE datasets MAARBLE (Monitoring, Analyzing and Assessing Radiation Belt Loss and Energisation) project will end in Dec 2014 Combined space and ground-based ULF wave database Spaceborne data are limited to distance of 3RE< R < 10 RE Within magnetic latitudes of ±60º Spaceborne VLF & ULF waves Cluster datasets use CAA datasets (EFW, FGM, STAFF) as inputs ULF/Pc3-5, ULF/Pc1-2, ULF/Pc1, VLF THEMIS (ULF/Pc3-5, ULF/Pc1-2, VLF) GOES (ULF/Pc3-5, ULF/Pc1-2) CHAMP (ULF/Pc3-5) Swarm (ULF/Pc3-5) Ground-based data CARISMA (ULF/Pc3-5) MIRACLE (ULF/Pc3-5)

MAARBLE - Wave parameters

ESOC Datasets

ESOC Archives ESOC datasets contain data as follows: Documents in pdf Digital data in xls or txt Some datasets are based on a single file that is updated annually ESOC datasets in 35 categories 6 documentation categories of miscellaneous sets of documents These will be placed in the normal CAA document area 29 categories contains a number of similar files (based on a template converted into pdf, xls or txt) each group handled as a single dataset, so they end up to 29 datasets (per spacecraft) Each category requires a separate conversion pipeline

ESOC Archive: L1 (Reports and Technical Notes) AR_Spacecraft Anomaly Reports CLU_AR-011 pdf CLU_AR-012 pdf … ARTS_Ground Segment Anomalies and Observation Reports CLU-1019 pdf CLU-1020 pdf Monthly Reports CLUSTER_Monthly_MOR_April_2001 pdf CLUSTER_Monthly_MOR_August_2001 pdf Weekly Mission Operations Reports Cluster_MOR_no022_(week01-2001) pdf Cluster_MOR_no023_(week02-2001) pdf Monthly/weekly reports will be ingested as NON-CEF DATASETS Anomaly reports will be made available under Documentation/ESOC

ESOC Archive: Anomaly Reports Timing of anomalies is non-trivial because it is not always clear when they started and ended, and most importantly, some of them are reoccurring Most likely user wants to find anomaly report according to other values than time Key path to an anomaly report is weekly reports where the anomalies are reported and discussed (and Anomaly Report Number is given): then one wishes to read the given anomaly report A simple xml table will be created for an access to anomaly reports (using the ESOC table underneath which shows the first few columns of their table)

ESOC Archive: L2 (BIT ERROR COUNTS) there is a single file covering all years and all s/c. It shall be explained (in the ICD) how the data are sampled and which correction was done: e.g. different sampling time (from AOS to AOS) and normalization to one word group. ESOC source file

ESOC Archive: L2 (SA Performance) there is a single file covering all years and all s/c [only digital data]

ESOC Archive: L2 (Eclipse) • information given in two different formats and documents released only until 2005; new format given for years 2005-2013

ESOC Archive: L2 (Manouvres)

Command History Files (from SPEVAL) The printout is made once a year (TBC), so Line number is a running number of command in the given year The file is saved as daily non-CEF files in order to keep the structure which helps reading the parameters If necessary, the dataset could be converted into CEF dataset where the last parameter gives all the parameters separated by semi-colon

User Statistics and User Interfaces Status of User Statistics and User Interfaces

User Statistics Usage of the system is monitored and the overview is displayed at http://caa.estec.esa.int/caa/user_stats.xml ~1900 different registered users ~200 different users every month No new users accepted Downloading most of the data are downloaded via the command-line interface Plenty of downloading via streaming interface total monthly downloading rate: 5-10 TeraByte. largest downloading rate is from February 2014 when ~9.9 TeraByte

Raw Data Delivery The separate RDM delivery service has continued to operate without problems CAA investigated the completeness of RDM data holdings and found four CD-roms with problems (either missing or files corrupted); ESOC is asked to investigate these CD-roms and provide them to the CAA The monthly splitting of the RDM downloading is as follows:

User Interfaces Users can have access to the CAA data via three interfaces; CSA has implemented the same interfaces Web interface very stable with only minor updates recently Command-line interface very stable with no changes recently. A few years ago the speed of command-line was improved significantly, so that this service can be used now in real-time display by the user graphical tools Streaming interface The streaming interface has been available for several years its usage has increased significantly in the past year In this service the data are not written on CAA disk, but are directly streamed to the user who can either forward the data directly to an application or write them into a file (and zip if requested).

Visualization Tools

Introduction CAA has developed four visualization tools Quick-look browsing tool Visualization of time series (both on-demand and pre-gen) Now CAA pre-generates only the daily panels used for the monitoring of the data quality Development of new panels continues for both Cluster and non-Cluster datasets such as Double Star and FP7 projects (ECLAT, MAARBLE) Visualization of particle distribution functions (only on-demand) Inventory plot browsing tool CSA visualization is based on the CAA scripts; 3rd tool is not yet implemented but will be done during summer 2014; implementation of the 1st tool is ongoing

Quick-Look Browsing Tool fast browsing tool, adopted from the CSDSweb CAA database contains plenty of quick-look plots: CSDSweb plots produced by UKDC based on raw data with default calibration values Instrument quick-look plots produced by the instrument teams produced at varying stages of data processing and do not necessarily represent data that are ingested into the CAA database. display various instrument auxiliary/housekeeping data that help users to evaluate the quality of the measurements Quick-look plots produced by the CAA based on the calibrated data ingested in the CAA database

Quick-Look Browsing Tool

Examples of New Overview Plots

CAA Visualization Tool for Time Series These are based on IDL scripts; CESR CEFLIB is used for reading the data into IDL pre-generated for three time intervals: 1-hr, 6-hr, and 24-hr User can arrange the order of the selected plots User can also save the selected panels and their order as a profile Multi-spacecraft tickbox added for line plots

CAA Visualization Tool for Time Series

CAA Visualization Tool for PDFs Following the recommendation made by the CAA review board in 2009, the CAA has implemented a visualization tool for the particle distribution functions The tool supports the following four plot types for each experiment (CIS, PEACE and RAPID) and particle type: Sauvaud: pitch angle shown for each energy bin and plotted against time Wheel or particle distribution plot (against pitch-angle directions) Angle-angle plot (in spacecraft coordinates, against latitude – longitude directions) Pitch-angle – energy plot Number of updates and corrections done during the past 12 months; several open action items still TBD

CAA Visualization Tool for PDFs All plots are produced on-demand only

Example: Sauvaud Plot

Example: Wheel Plot

CAA Cross-Calibration Activity since June 2013

CAA Cross-Calibration Workshops

CAA Special Issue in EGU GI Journal CAA Calibration Results will appear in Geoscientific Instrumentation, Methods and Data Systems (GI). This is peer-reviewed free online access [as is the case for all EGU journals] normal scientific journal, so please avoid review-type articles (no page limit however) but split your calibration/cross-calibration results into separate articles All papers that deal with calibration, cross-calibration and data processing issues of the Cluster and Double Star instrumentation are welcome NOTE: GI has one special issue per year, and they selected the Cluster calibration results as the topic of the 2013 special issue Why a Special Issue CAA calibration/cross-calibration activity is quite unique in experimental space physics and the results are worth of publishing Literature on instrument calibration methods and long-term results is very limited

CAA Special Issue in EGU GI Journal http://www.geosci-instrum-method-data-syst-discuss.net/special_issue7.html

18th Cross-Calibration Workshop Hosted by the CAA/STFC team and held Cosener’s House, Abingdon, UK, on 23-25 October 2013 21 participants and 29 presentations 16 action items were raised 12 action items are now closed 4 are still open

18th Cross-Calibration Workshop

18th Cross-Calibration Workshop

19th Cross-Calibration Workshop Held at at ESRIN, Frascati, Italy, on 2-4 April 2014 22 participants and 33 presentations 23 action items were raised that are still formally open

19th Cross-Calibration Workshop

19th Cross-Calibration Workshop

Future Development of the CAA

Continuing Activities Switch-over to CSA for user access to the Cluster archive CAA is expected to close to public in June 2014 CSA usage does not increase before CAA is closed Currently the Cluster mission is scheduled to end in December 2014 Note that there is always one additional year for the completion of the CAA activities SPC decides about the extension of Cluster mission for another two years (2015-2016) in November 2014 Routine operations, data delivery and maintenance of the CAA database continues as before Ongoing activities of the Cross-Calibration Working Group Support for non-Cluster product generation and ingestion, e.g., Double Star, ECLAT and MAARBLE Complete the production of all ESOC datasets; update of these datasets happen once a year Production of various science and auxiliary datasets Updates in visualization tools

New Developments Production and development of new products Generation of new and auxiliary science datasets Process JSOC datasets, produce the metadata and ingest the files to the database Development of value-added products such as event lists, derived quantities and products in support of the cross-calibration and quality assessment activities: Lists of boundary crossings and regions Visualization/graphical products Production of survey data products (used particularly in data mining) New cross-calibration activities New visualization tools New software New cefpass development in java

CAA-CSA Interface Matters Work for the CSA routine data and metadata transfer from the CAA to the CSA Generation and transfer of inventory files technical and scientific support for the datasets continued development and extension of the visualization tools After the switch-over to CSA, the remaining user services in the CAA RDM distribution Command-line access to the file listing from the CAA database (for the data producers)