1. Overview of the CAA Activities
H. Laakso

2. Summary of Activities
CAA development activities in 2013/4 have primarily focused on
maintenance and optimization of the existing core system
support for the CAA to CSA migration
updating of visualization tools
production of cross-calibration comparisons
Development of new software for the Double Star datasets; EFW science datasets; velocity moments in inertial frame in both GSE and ISR2; bow shock crossings/solar wind intervals
production of new auxiliary datasets, including ESOC datasets
Interaction on deliveries of data from non-Cluster sources:
Double Star - ongoing
EU FP-7 ECLAT (European Cluster Assimilation Technology) - completed
MAARBLE (Monitoring, Analyzing and Assessing Radiation Belt Loss and Energization) - ongoing

3. Status of File Transfer to CSA

4. Status of CAA Hardware

5. Management and Standards

6. Management

7. Cross-Calibration Workshops
The organization of regular cross-calibration workshops, even at this stage in the mission, has been a vital factor in the successful implementation of the CAA and this is an important strategic lesson for any space science project in the future.
Two cross-calibration workshops were held since the last review (October 2013 in Abingdon, and April 2014 in Frascati)
An overview of these meetings is given in document CAA-EST-RP-0102
The minutes of the meetings with all presentations can be found at
In addition the minutes of the meetings from these two workshops are attached to the review package as reference documents

8. Instrument Progress Meetings
An annual progress meeting was held between CAA and each team during Nov June A normal agenda of the progress meetings is
1. Welcome
2. Status of the instrument datasets in the CAA
3. Status of Action Items
a. Status of action items from the Progress Meetings
b. Status of action items from the Cross-Calibration Workshops
c. Status of the OR Recommendations
Status of data production and delivery
Status of Calibration and cross-calibration activities
Status of Double Star archiving
6. Status of Documentation (UG, CR, ICD)
7. AOB
The minutes of the progress meetings are attached to the review package
the minutes of the meetings with all presentations can be found at ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/ProgressMeetings/

9. Management of Action Items
Due to a large number of action items from Instrument Progress Meetings and Cross-Calibration Workshops, the management of action items has been challenging.
To help their management a new GUI was developed. The page can be accessed at

10. CAA Metadata Dictionary, v3.0
New version, v3.0, can be downloaded at CAA/Documentation
Previous release, v2.3, was published on 5 March 2008 and is outdated
New document name (CAA-MDD-0001) because MDD is now maintained by the CAA
Style is a bit simplified & changed in order to ease its reading & browsing
Key updates
Several updates in enumerated values for the following keywords:
MISSION, MISSION_AGENCY, EXPERIMENT, OBSERVATORY, INSTRUMENT_NAME for the Double Star mission
New values for MEASUREMENT_TYPE, INSTRUMENT_TYPE, PROPERTY, TENSOR_FRAME
Modified definition for QUALITY
New coordinate systems: FAC, MFA
new keyword: TARGET_SYSTEM
Changes in appendices
Appendix D deleted
Enumerated lists are now given in Appendix B

11. Update of the CAA Metadata Dictionary, v3.0

12. Dataset Production, Delivery and Ingestion
Status of
Dataset Production, Delivery and Ingestion

13. Cluster Datasets – Delivery Status
The baseline of the data delivery is to provide all 2013 data by the 9th Operations Review
Various challenges exist such as
there are either difficult calibration issues with aging instruments, or
there are important instrument dependencies in the data production

14. Cluster datasets - Critical Datasets
There are four critical datasets that are needed by many other teams for their data processing and calibration activities
Realistically one cannot expect the data deliveries sooner than ~6 months after the delivery of these critical datasets.
The delivery status of the critical datasets is :
DWP time correction: up to Dec 2013
EFW spacecraft potential: up to Sep 2013
FGM science datasets: up to Feb 2014
WHISPER density: up to July 2013

15. New Datasets: Velocity Moments
Created for PEACE and CIS
Parallel and Perpendicular Velocity in inertial frame in both GSE and ISR2
(spin-averaged) DC Electric Field in inertial frame in both GSE and ISR2 (based on PerpVelocity)
The magnetic field used in all calculations is FGM_5VPS from which the magnetic field is determined for the time interval of the given velocity moment
The parameters in the new velocity datasets are:
Data_Velocity_GSE_inert
Data_Velocity_ISR2_inert
Data_PerpVelocity_GSE_inert
Data_PerpVelocity_ISR2_inert
Data_ParVelocity_GSE_inert
Data_ParVelocity_ISR2_inert
Data_Efield_GSE_inert
Data_Efield _ISR2_inert
Hourly comparisons between EFW, EDI, PEACE and CIS created for DC E-fields and Drift Velocities
Plots can be downloaded or viewed via QL Browsing Tool
These variables can be compared with EDI and EFW

16. DC E-field at magnetopause

17. C1: Drift Velocity in the plasma sheet

18. C3: Drift Velocity in the plasma sheet

19. Detection of Boundary Crossings
The work on the detection of the boundary crossings by Cluster is at its final stage
Data processing and detection is done with C
Plotting is done with matlab
The first boundary to be determined is the Bow Shock
Required step-like changes must happen in both Bt and HIA Vx within a few minutes if they available
If only one of them is available, then two other steps are required (several parameters are investigated)
Other boundaries:
The following work will be the radiation belt crossings which is expected to happen quickly
The magnetopause crossings will be the last one
User recommendation:
it is better to generate list of region intervals rather than the boundary crossings, e.g. list of intervals when the spacecraft is in the solar wind
This is not trivial as the spacecraft can be off during a bow shock crossing

20. Example: Bow Shock Crossings
Script is working quite well but human validation is needed in order to add new boundaries and remove wrong boundaries

21. EU/FP7 ECLAT datasets
The EU FP7 ECLAT (European Cluster Assimilation Technology) project was completed in Feb 2014
ECLAT datasets are being transferred to the CSA

22. ECLAT
Quicklook Plots
ECLAT Quick-look Plots

23. EU/FP7 MAARBLE datasets
MAARBLE (Monitoring, Analyzing and Assessing Radiation Belt Loss and Energisation) project will end in Dec 2014
Combined space and ground-based ULF wave database
Spaceborne data are limited to distance of 3RE< R < 10 RE
Within magnetic latitudes of ±60º
Spaceborne VLF & ULF waves
Cluster datasets use CAA datasets (EFW, FGM, STAFF) as inputs
ULF/Pc3-5, ULF/Pc1-2, ULF/Pc1, VLF
THEMIS (ULF/Pc3-5, ULF/Pc1-2, VLF)
GOES (ULF/Pc3-5, ULF/Pc1-2)
CHAMP (ULF/Pc3-5)
Swarm (ULF/Pc3-5)
Ground-based data
CARISMA (ULF/Pc3-5)
MIRACLE (ULF/Pc3-5)

24. MAARBLE - Wave parameters

25. ESOC Datasets

26. ESOC Archives ESOC datasets contain data as follows: Documents in pdf
Digital data in xls or txt
Some datasets are based on a single file that is updated annually
ESOC datasets in 35 categories
6 documentation categories of miscellaneous sets of documents
These will be placed in the normal CAA document area
29 categories contains a number of similar files (based on a template converted into pdf, xls or txt)
each group handled as a single dataset, so they end up to 29 datasets (per spacecraft)
Each category requires a separate conversion pipeline

27. ESOC Archive: L1 (Reports and Technical Notes)
AR_Spacecraft Anomaly Reports
CLU_AR-011 pdf
CLU_AR-012 pdf …
ARTS_Ground Segment Anomalies and Observation Reports
CLU-1019 pdf
CLU-1020 pdf
Monthly Reports
CLUSTER_Monthly_MOR_April_2001 pdf
CLUSTER_Monthly_MOR_August_2001 pdf
Weekly Mission Operations Reports
Cluster_MOR_no022_(week ) pdf
Cluster_MOR_no023_(week ) pdf
Monthly/weekly reports will be ingested as NON-CEF DATASETS
Anomaly reports will be made available under Documentation/ESOC

28. ESOC Archive: Anomaly Reports
Timing of anomalies is non-trivial because it is not always clear when they started and ended, and most importantly, some of them are reoccurring
Most likely user wants to find anomaly report according to other values than time
Key path to an anomaly report is weekly reports where the anomalies are reported and discussed (and Anomaly Report Number is given): then one wishes to read the given anomaly report
A simple xml table will be created for an access to anomaly reports (using the ESOC table underneath which shows the first few columns of their table)

29. ESOC Archive: L2 (BIT ERROR COUNTS)
there is a single file covering all years and all s/c.
It shall be explained (in the ICD) how the data are sampled and which correction was done: e.g. different sampling time (from AOS to AOS) and normalization to one word group.
ESOC source file

30. ESOC Archive: L2 (SA Performance)
there is a single file covering all years and all s/c [only digital data]

31. ESOC Archive: L2 (Eclipse)
• information given in two different formats and documents released only until 2005; new format given for years

32. ESOC Archive: L2 (Manouvres)

33. Command History Files (from SPEVAL)
The printout is made once a year (TBC), so Line number is a running number of command in the given year
The file is saved as daily non-CEF files in order to keep the structure which helps reading the parameters
If necessary, the dataset could be converted into CEF dataset where the last parameter gives all the parameters separated by semi-colon

34. User Statistics and User Interfaces
Status of
User Statistics and User Interfaces

35. User Statistics
Usage of the system is monitored and the overview is displayed at
~1900 different registered users
~200 different users every month
No new users accepted
Downloading
most of the data are downloaded via the command-line interface
Plenty of downloading via streaming interface
total monthly downloading rate: 5-10 TeraByte.
largest downloading rate is from February 2014 when ~9.9 TeraByte

36. Raw Data Delivery
The separate RDM delivery service has continued to operate without problems
CAA investigated the completeness of RDM data holdings and found four CD-roms with problems (either missing or files corrupted); ESOC is asked to investigate these CD-roms and provide them to the CAA
The monthly splitting of the RDM downloading is as follows:

37. User Interfaces
Users can have access to the CAA data via three interfaces; CSA has implemented the same interfaces
Web interface
very stable with only minor updates recently
Command-line interface
very stable with no changes recently.
A few years ago the speed of command-line was improved significantly, so that this service can be used now in real-time display by the user graphical tools
Streaming interface
The streaming interface has been available for several years
its usage has increased significantly in the past year
In this service the data are not written on CAA disk, but are directly streamed to the user who can either forward the data directly to an application or write them into a file (and zip if requested).

38. Visualization Tools

39. Introduction CAA has developed four visualization tools
Quick-look browsing tool
Visualization of time series (both on-demand and pre-gen)
Now CAA pre-generates only the daily panels used for the monitoring of the data quality
Development of new panels continues for both Cluster and non-Cluster datasets such as Double Star and FP7 projects (ECLAT, MAARBLE)
Visualization of particle distribution functions (only on-demand)
Inventory plot browsing tool
CSA visualization is based on the CAA scripts; 3rd tool is not yet implemented but will be done during summer 2014; implementation of the 1st tool is ongoing

40. Quick-Look Browsing Tool
fast browsing tool, adopted from the CSDSweb
CAA database contains plenty of quick-look plots:
CSDSweb plots produced by UKDC
based on raw data with default calibration values
Instrument quick-look plots produced by the instrument teams
produced at varying stages of data processing and do not necessarily represent data that are ingested into the CAA database.
display various instrument auxiliary/housekeeping data that help users to evaluate the quality of the measurements
Quick-look plots produced by the CAA
based on the calibrated data ingested in the CAA database

41. Quick-Look Browsing Tool

42. Examples of New Overview Plots

43. CAA Visualization Tool for Time Series
These are based on IDL scripts; CESR CEFLIB is used for reading the data into IDL
pre-generated for three time intervals: 1-hr, 6-hr, and 24-hr
User can arrange the order of the selected plots
User can also save the selected panels and their order as a profile
Multi-spacecraft tickbox added for line plots

44. CAA Visualization Tool for Time Series

45. CAA Visualization Tool for PDFs
Following the recommendation made by the CAA review board in 2009, the CAA has implemented a visualization tool for the particle distribution functions
The tool supports the following four plot types for each experiment (CIS, PEACE and RAPID) and particle type:
Sauvaud: pitch angle shown for each energy bin and plotted against time
Wheel or particle distribution plot (against pitch-angle directions)
Angle-angle plot (in spacecraft coordinates, against latitude – longitude directions)
Pitch-angle – energy plot
Number of updates and corrections done during the past 12 months; several open action items still TBD

46. CAA Visualization Tool for PDFs
All plots are produced on-demand only

47. Example: Sauvaud Plot

48. Example: Wheel Plot

49. CAA Cross-Calibration Activity since June 2013

50. CAA Cross-Calibration Workshops

51. CAA Special Issue in EGU GI Journal
CAA Calibration Results will appear in Geoscientific Instrumentation, Methods and Data Systems (GI). This is
peer-reviewed
free online access [as is the case for all EGU journals]
normal scientific journal, so please avoid review-type articles (no page limit however) but split your calibration/cross-calibration results into separate articles
All papers that deal with calibration, cross-calibration and data processing issues of the Cluster and Double Star instrumentation are welcome
NOTE: GI has one special issue per year, and they selected the Cluster calibration results as the topic of the 2013 special issue
Why a Special Issue
CAA calibration/cross-calibration activity is quite unique in experimental space physics and the results are worth of publishing
Literature on instrument calibration methods and long-term results is very limited

52. CAA Special Issue in EGU GI Journal

53. 18th Cross-Calibration Workshop
Hosted by the CAA/STFC team and held Cosener’s House, Abingdon, UK, on October 2013
21 participants and 29 presentations
16 action items were raised
12 action items are now closed
4 are still open

54. 18th Cross-Calibration Workshop

55. 18th Cross-Calibration Workshop

56. 19th Cross-Calibration Workshop
Held at at ESRIN, Frascati, Italy, on 2-4 April 2014
22 participants and 33 presentations
23 action items were raised that are still formally open

57. 19th Cross-Calibration Workshop

58. 19th Cross-Calibration Workshop

59. Future Development of the CAA

60. Continuing Activities
Switch-over to CSA for user access to the Cluster archive
CAA is expected to close to public in June 2014
CSA usage does not increase before CAA is closed
Currently the Cluster mission is scheduled to end in December 2014
Note that there is always one additional year for the completion of the CAA activities
SPC decides about the extension of Cluster mission for another two years ( ) in November 2014
Routine operations, data delivery and maintenance of the CAA database continues as before
Ongoing activities of the Cross-Calibration Working Group
Support for non-Cluster product generation and ingestion, e.g., Double Star, ECLAT and MAARBLE
Complete the production of all ESOC datasets; update of these datasets happen once a year
Production of various science and auxiliary datasets
Updates in visualization tools

61. New Developments Production and development of new products
Generation of new and auxiliary science datasets
Process JSOC datasets, produce the metadata and ingest the files to the database
Development of value-added products such as event lists, derived quantities and products in support of the cross-calibration and quality assessment activities:
Lists of boundary crossings and regions
Visualization/graphical products
Production of survey data products (used particularly in data mining)
New cross-calibration activities
New visualization tools
New software
New cefpass development in java

62. CAA-CSA Interface Matters
Work for the CSA
routine data and metadata transfer from the CAA to the CSA
Generation and transfer of inventory files
technical and scientific support for the datasets
continued development and extension of the visualization tools
After the switch-over to CSA, the remaining user services in the CAA
RDM distribution
Command-line access to the file listing from the CAA database (for the data producers)