1. Cluster Active Archive Overview
Chris Perry
CAA Technical Manager
Implementation Review
ESTEC
31st May 2005

2. CAA Implementation Review
Introduction
Overview
Management – staff/contracts/activities
Standards – data formats and metadata
Products – interface control document meetings
Infrastructure – hardware and network
Summary
CAA Implementation Review
31st May 2005

3. CAA Implementation Review
CAA Overview
The scientific rationale for the ESA Cluster Active Archive:
“Maximise the scientific return from the mission by making all Cluster data available to the world-wide scientific community.”
“Ensure that the unique data set returned by the Cluster mission is preserved in a stable, long-term archive for scientific analysis beyond the end of the mission.”
“Provide this archive as a major contribution by ESA and the Cluster science community to the International Living With a Star programme.”
CAA Implementation Review
31st May 2005

4. CAA Implementation Review
Overview
What does this mean?
The CAA should contain (all) the Cluster high resolution data.
The data should be of the best (achievable) quality.
The data should be suitable for detailed science investigations.
Need to start now while expertise still available.
ESA providing support to instrument teams to ensure this.
In addition the CAA will hold ancillary products and support information including
Auxiliary data (such as orbit and attitude)
Survey data and plots
CSDS data including JSOC parameters
Documentation
Others…?
CAA Implementation Review
31st May 2005

5. CAA Implementation Review
CAA Structure – Team
The full CAA team is now mainly in place…
Project Manager (
Trevor Sanderson )
Technical
Manager
Chris Perry
System Engineer
Steve
Esson
Harald
Jeszensky
Klaus
Torkar
Wave Consortium
JSOC
CIS
DWP
ASPOC
PEACE
EFW
EDI
WBD
FGM
STAFF
WHISPER
RAPID
Archive Developer
Sinead McCaffrey
Alain
Barthe
Henri
Reme
Edita
Georgescu
Berndt
Klecker
TBC
Andrew
Fazakerley
Stefan
Mühlbachler
Patrick Daly
Tobias Eriksson
Biagio
Forte
Andre
Balogh
Hugo
Alleyne
Yuri
Khotyaintsev P. - A.
Lindqvist
Mats Andre
Laurent
Mirioni
Nicole C
Wehrlin
Gungör
Bozan
Pierrette
Decreau
Cluster Project
Scientist
Phillipe
Escoubet
Deputy
Harri
Laakso
ESOC
Rob Wilson (temp)
Jonny
Gloag
Jolene Pickett
Working Group
C. Harvey
A. Allen
CAA Implementation Review
31st May 2005

6. CAA Implementation Review
CAA Structure - Contracts
Instrument team contracts are in place…
Instrument
Technical Contact
Contract Start
Contract Sign
ASPOC
Klaus Torkar
Harald Jeszensky
1/2/2004
15/4/2004
CIS
Henri Reme
Iannis Dandourous
Alain Barthe
Emmanuel Penou
1/3/2004
DWP
Hugo Alleyne
tba
5/2005
5/2005 tbc
EDI
Berndt Klecker
Edita Georgescu
21/4/2004
EFW
Mats Andre
Yuri Khotyaintsev
P.-A. Lindqvist
25/3/2004
FGM
Andre Balogh
Jonny Gloag
1/6/2004
2/6/2004
PEACE
Andrew Fazakerley
Rob Wilson (temp)
9/2004
RAPID
Patrick Daly
Stefan Mühlbachler
STAFF
Nicole Cornilleau
Laurent Mirioni
WBD
n/a
Jolene Pickett
WHISPER
Pierrette Decreau
Gungör Bozan
22/4/2004
ESOC
Jurgen Volpp
David Fornarelli
CAA Implementation Review
31st May 2005

7. CAA Implementation Review
A lot has been happening…
CAA Structure - Activities
2004
2005
Reviews
Core team
System dev.
Standards
Reporting
Product dev.
CAA Implementation Review
31st May 2005

8. CAA Implementation Review
Standards
The CAA Metadata working group (led by Chris Harvey, CDPP) recently released v2.0 of the CAA Metadata Dictionary (CAA-CDPP-TN-0002).
Metadata provides a standard machine readable way of storing descriptions of the data.
This will aid the consistent location and handling of all data products and in particular the digital parameter (science) data.
Much iteration between instrument teams, developers and CH. Current version is largely stable but some open issues remain.
Interoperability issues being addressed in conjunction with SPASE (Space Physics Search and Exchange).
CAA Implementation Review
31st May 2005

9. Standards – Digital Parameter Data
Instrument teams will deliver digital parameter data using
one of two formats:-
ASCII: Cluster Exchange Format (DS-QMW-TN-0010)
or for legacy data:-
Binary: Common Data Format (DS-QMW-TN-0003)
The CAA data formats working group (led by Tony Allen, IC) has addressed updates to CEF (new CEF v2.0 specification) to support development of CAA datasets.
The CAA shall allow users to download data in at least CEF 2 and CDF.
Conceptually the files making up a dataset are treated as a single time series.
CAA Implementation Review
31st May 2005

10. CAA Implementation Review
CAA Information Systems
CAA web site:
From there see links to:
Technical Forum:
LiveLink: (requires login)
CAA Implementation Review
31st May 2005

11. CAA Implementation Review
Products Development
The detailed description of instrument team products is specified in the instrument-to-CAA interface control document (ICD).
The CAA core team has been visiting each of the instrument teams to assist in the specification of the products to be supplied.
The meetings have also been used to identify initial products that shall be delivered for testing.
Instrument teams progressing at different rates but major effort made by all teams to supply inputs for the coming review.
Need to ensure that this good progress is maintained.
CAA Implementation Review
31st May 2005

12. CAA Implementation Review
CAA – Delivery
Overview of instrument deliveries for 2001
CAA Implementation Review
31st May 2005

13. CAA Implementation Review
CAA – Wave Instrument Status
TEAM
ICD
Updated
Data Products
Interval Provided
DWP
04-May
TCOR, LOG, COR, UT_PIOR
Test data only
EFW
23-Apr
L1_P12, L1_P1, L1_P2, L1_P34, L1_P3, L1_P4, L2_E, L2_P, L3_E, L3_P ,
52 Datasets, 1720 files
FGM
22-Apr
FGM_FULL, FGM_5VPS, FGM_SPIN, FGM_CALF, FGM_CALA, FGM_CAVF, FGM_GAPF, FGM_VALF
>
20 Datasets, 82 files
STAFF
09-May
DWF_HBR, DWF_NBR, AGC, PSD, SM, spectrograms,
>
8 Datasets, 1694 files
WBD
Spectrogram plots
>
WHISPER
NATURAL, WAVE_FORM_ENERGY, ACTIVE, PASSIVE, ACTIVE_TO_PASSIVE_RATIO, HK (Sounding), ACTIVE_EVENT, NATURAL_EVENT, ELECTRON_DENSITY
>
32 datasets, 2167 files
CAA Implementation Review
31st May 2005

14. CAA Implementation Review
CAA – Particle Instrument Status
TEAM
ICD
Updated
Data Products
Interval Provided
ASPOC
Issue 1.1
11-May
STAT, IONC, IONS, CMDH
>
20 Dataset, 2602 files
CIS
Issue 1.0
04-May
HIA: IONS_CS,IONS_PEF, IONS_PF,IONS_RC
CODIF; CODIF_HS_MAG_H1_CS, CODIF_HS_MAG_H1_PEF, CODIF_HS_MAG_H1_PE
>
32 Datasets, 1092 files
EDI
Draft 0.2
06-May
DOC, EGD, CAVEATS, MPD, MSF, PP, Ppplus, QSTAT
>
8 Dataset, 742 files
PEACE
02-May
Many, see ICD
>
40 Datasets, 4199 files
RAPID
>
88 Datasets, 8793 files
CAA Implementation Review
31st May 2005

15. CAA Implementation Review
CAA – Other Products
ESOC Products
Diverse range of information including:
Reports and Technical Notes, Spacecraft System Data, Mission Planning System Data, Flight Dynamics Data, Ground Segment Data, Graphical Products
Delivered to CAA in native format (mostly Excel & Word)
CAA will be responsible for transcription to pdf and production of metadata.
A small sub-set of Excel products to be converted to CAA digital data sets/event lists.
Products delivered for 2004
Prime, Summary and JSOC Parameter Products
Available as validated, self describing CDF files from CSDS
Conversion to CEF possible via Qtran
Useful as a quality check for full resolution products
CAA Implementation Review
31st May 2005

16. CAA Implementation Review
CAA RDM Delivery
For extended mission RDM CD-ROM delivery from ESOC will cease
CAA will provide RDM data accessible over the network.
DDS access will continue unchanged.
Meeting held with ESOC to discuss I/F between ESOC and CAA.
Testing confirmed that network delivery, ESOC to CAA, is viable.
CD-ROM directory structure will be maintained.
Test transfer of 650 MB best case of 4 mins, 2.4 MB/s (uncompressed)
Details of CAA to community interface need to be clarified.
Delivery of full directory structure (Zip, Rsync etc.).
Or user select parts of the RDM directory structure that they want (e.g. by instrument) and just these get delivered.
Discussed at IWG earlier this month, document outlining options to be distributed to instrument teams and data centres next week.
CAA Implementation Review
31st May 2005

17. CAA Implementation Review
CAA Infrastructure - Status
CAA Hardware Status
Room at ESTEC has been refurbished (air con etc)
System hardware was delivered during August 2004
Basic configuration (operating system and most third party software, CVS, web server etc) is now complete.
System is running Debian version of Linux. (not SUSE)
Some problems encountered with high reliability component (failover) of this high reliability system!
High speed network connection via ESAGrid (1 Gbit/s).
Development of CAA data ingestion and cataloguing components.
Basic web access capability in place.
CAA Implementation Review
31st May 2005

18. CAA Implementation Review
CAA Infrastructure – System/Network
System
INTERNET s u r f n e t E Q U A N T
ESAgrid R
ESTEC Front -
End network
CAA1
CAA2
ESOC
RAL
770 -
orleans.fr
WHISPER
200
msslx.mssl.ucl.ac.uk
PEACE
850
sun2.mpae.gwdg.de
RAPID
EFW
120
cassini.physics.uiowa.edu
WBD 95
cis.cesr.fr
CIS
250
rhea.iwf.oeaw.ac.at
ASPOC
750
mpe.mpg.de
EDI
Average
Speed KB/s
Address
Data
msslhx.mssl.ucl.ac.uk
650
cddsa.esoc.ops.esa.int 90
Network
CAA Implementation Review
31st May 2005

19. CAA Implementation Review
CAA Infrastructure – Software Development
Conceptual design unchanged from System Specification Review
Development activities concentrated on data ingestion and cataloguing.
Prototype ingestion system in place, tested with more than 20k test and delivered CEF files.
Basic Web data access system allows simple queries and retrieval of packaged data.
Third party products such as QSAS updated to support CAA products.
CAA Implementation Review
31st May 2005

20. CAA Implementation Review
CAA Infrastructure – Web Access
Initial login
Top level of “Basic” search
CAA Implementation Review
31st May 2005

21. CAA Implementation Review
CAA – QSAS support for CEF-2 C4 C2 C3 C1
1 minute
CAA Implementation Review
31st May 2005

22. CAA Implementation Review
CAA Future Planning
2005
2006
Reviews ♦
Data
S/W Dev.
H/W Dev.
RDM Deliv.
CAA Implementation Review
31st May 2005

23. CAA Implementation Review
Summary
After a delayed start-up the CAA activities are making good progress.
The CAA standards work provides a firm basis on which to build the high quality science products.
A highly productive set of meetings have been held with the instrument teams to define the delivered products.
Delivery of Level 1, initial science data and some ancillary products (e.g. plot files) to the CAA is underway.
A basic system to allow selection and retrieval of data is in place
The details of these activities will be provided in the following presentations.
CAA Implementation Review
31st May 2005

24. CAA Implementation Review
Review Presentations
Tuesday Afternoon
Data Format Working Group Report Tony Allen
Data Model and Data Dictionary WG Reoprt Chris Harvey
Standards tools – CAA tools Tobias Eriksson
14:15 Coffee
Fields and Waves Instrument Status Tobias Eriksson
DWP Keith Yearby
EFW Yuri Khotyaintsev
FGM Jonathan Gloag
STAFF Laurent Mirioni
WHISPER Bozan Güngör
WBD (TBC)
End
CAA Implementation Review
31st May 2005

25. CAA Implementation Review
Review Presentations
Wednesday
08:45 Start
Overview of particles status Sinead McCaffrey
ASPOC Harald Jeszenszky
CIS Alain Barthe
EDI Edita Georgescu
PEACE Rob Wilson
RAPID Stefan Mühlbachler
10:35 Coffee
ESOC (TBC)
CAA System Infrastructure Steve Esson
CAA Software Development Sinead McCaffrey
CAA future development plans, next steps Chris Perry
12:00 Summary/Q & A (All)
12:30 End of open session
CAA Implementation Review
31st May 2005