Cluster Active Archive Overview Chris Perry CAA Technical Manager Implementation Review ESTEC 31st May 2005

CAA Implementation Review Introduction Overview Management – staff/contracts/activities Standards – data formats and metadata Products – interface control document meetings Infrastructure – hardware and network Summary CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Overview The scientific rationale for the ESA Cluster Active Archive: “Maximise the scientific return from the mission by making all Cluster data available to the world-wide scientific community.” “Ensure that the unique data set returned by the Cluster mission is preserved in a stable, long-term archive for scientific analysis beyond the end of the mission.” “Provide this archive as a major contribution by ESA and the Cluster science community to the International Living With a Star programme.” CAA Implementation Review 31st May 2005

CAA Implementation Review Overview What does this mean? The CAA should contain (all) the Cluster high resolution data. The data should be of the best (achievable) quality. The data should be suitable for detailed science investigations. Need to start now while expertise still available. ESA providing support to instrument teams to ensure this. In addition the CAA will hold ancillary products and support information including Auxiliary data (such as orbit and attitude) Survey data and plots CSDS data including JSOC parameters Documentation Others…? CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Structure – Team The full CAA team is now mainly in place… Project Manager ( Trevor Sanderson ) Technical Manager Chris Perry System Engineer Steve Esson Harald Jeszensky Klaus Torkar Wave Consortium JSOC CIS DWP ASPOC PEACE EFW EDI WBD FGM STAFF WHISPER RAPID Archive Developer Sinead McCaffrey Alain Barthe Henri Reme Edita Georgescu Berndt Klecker TBC Andrew Fazakerley Stefan Mühlbachler Patrick Daly Tobias Eriksson Biagio Forte Andre Balogh Hugo Alleyne Yuri Khotyaintsev P. - A. Lindqvist Mats Andre Laurent Mirioni Nicole C Wehrlin Gungör Bozan Pierrette Decreau Cluster Project Scientist Phillipe Escoubet Deputy Harri Laakso ESOC Rob Wilson (temp) Jonny Gloag Jolene Pickett Working Group C. Harvey A. Allen CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Structure - Contracts Instrument team contracts are in place… Instrument Technical Contact Contract Start Contract Sign ASPOC Klaus Torkar Harald Jeszensky 1/2/2004 15/4/2004 CIS Henri Reme Iannis Dandourous Alain Barthe Emmanuel Penou 1/3/2004 DWP Hugo Alleyne tba 5/2005 5/2005 tbc EDI Berndt Klecker Edita Georgescu 21/4/2004 EFW Mats Andre Yuri Khotyaintsev P.-A. Lindqvist 25/3/2004 FGM Andre Balogh Jonny Gloag 1/6/2004 2/6/2004 PEACE Andrew Fazakerley Rob Wilson (temp) 9/2004 RAPID Patrick Daly Stefan Mühlbachler STAFF Nicole Cornilleau Laurent Mirioni WBD n/a Jolene Pickett WHISPER Pierrette Decreau Gungör Bozan 22/4/2004 ESOC Jurgen Volpp David Fornarelli CAA Implementation Review 31st May 2005

CAA Implementation Review A lot has been happening… CAA Structure - Activities 2004 2005 Reviews Core team System dev. Standards Reporting Product dev. CAA Implementation Review 31st May 2005

CAA Implementation Review Standards The CAA Metadata working group (led by Chris Harvey, CDPP) recently released v2.0 of the CAA Metadata Dictionary (CAA-CDPP-TN-0002). Metadata provides a standard machine readable way of storing descriptions of the data. This will aid the consistent location and handling of all data products and in particular the digital parameter (science) data. Much iteration between instrument teams, developers and CH. Current version is largely stable but some open issues remain. Interoperability issues being addressed in conjunction with SPASE (Space Physics Search and Exchange). CAA Implementation Review 31st May 2005

Standards – Digital Parameter Data Instrument teams will deliver digital parameter data using one of two formats:- ASCII: Cluster Exchange Format (DS-QMW-TN-0010) or for legacy data:- Binary: Common Data Format (DS-QMW-TN-0003) The CAA data formats working group (led by Tony Allen, IC) has addressed updates to CEF (new CEF v2.0 specification) to support development of CAA datasets. The CAA shall allow users to download data in at least CEF 2 and CDF. Conceptually the files making up a dataset are treated as a single time series. CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Information Systems CAA web site: http://caa.estec.esa.int/ From there see links to: Technical Forum: http://www.cluster.rl.ac.uk/caa-bin/yabb/YaBB.pl LiveLink: http://www.rssd.esa.int/llink/livelink (requires login) CAA Implementation Review 31st May 2005

CAA Implementation Review Products Development The detailed description of instrument team products is specified in the instrument-to-CAA interface control document (ICD). The CAA core team has been visiting each of the instrument teams to assist in the specification of the products to be supplied. The meetings have also been used to identify initial products that shall be delivered for testing. Instrument teams progressing at different rates but major effort made by all teams to supply inputs for the coming review. Need to ensure that this good progress is maintained. CAA Implementation Review 31st May 2005

CAA Implementation Review CAA – Delivery Overview of instrument deliveries for 2001 CAA Implementation Review 31st May 2005

CAA Implementation Review CAA – Wave Instrument Status TEAM ICD Updated Data Products Interval Provided DWP 04-May TCOR, LOG, COR, UT_PIOR Test data only EFW 23-Apr L1_P12, L1_P1, L1_P2, L1_P34, L1_P3, L1_P4, L2_E, L2_P, L3_E, L3_P 20010518, 20010601 52 Datasets, 1720 files FGM 22-Apr FGM_FULL, FGM_5VPS, FGM_SPIN, FGM_CALF, FGM_CALA, FGM_CAVF, FGM_GAPF, FGM_VALF 20010130->20010216 20 Datasets, 82 files STAFF 09-May DWF_HBR, DWF_NBR, AGC, PSD, SM, spectrograms, 20010102->20011231 8 Datasets, 1694 files WBD Spectrogram plots 20010201->20011231 WHISPER NATURAL, WAVE_FORM_ENERGY, ACTIVE, PASSIVE, ACTIVE_TO_PASSIVE_RATIO, HK (Sounding), ACTIVE_EVENT, NATURAL_EVENT, ELECTRON_DENSITY 20010201->20010430 32 datasets, 2167 files CAA Implementation Review 31st May 2005

CAA Implementation Review CAA – Particle Instrument Status TEAM ICD Updated Data Products Interval Provided ASPOC Issue 1.1 11-May STAT, IONC, IONS, CMDH 20010102-> 20010701 20 Dataset, 2602 files CIS Issue 1.0 04-May HIA: IONS_CS,IONS_PEF, IONS_PF,IONS_RC CODIF; CODIF_HS_MAG_H1_CS, CODIF_HS_MAG_H1_PEF, CODIF_HS_MAG_H1_PE 20010201->20010228 32 Datasets, 1092 files EDI Draft 0.2 06-May DOC, EGD, CAVEATS, MPD, MSF, PP, Ppplus, QSTAT 20010109 -> 20020101 8 Dataset, 742 files PEACE 02-May Many, see ICD 20010201 -> 20010731 40 Datasets, 4199 files RAPID 20010101 -> 20010701 88 Datasets, 8793 files CAA Implementation Review 31st May 2005

CAA Implementation Review CAA – Other Products ESOC Products Diverse range of information including: Reports and Technical Notes, Spacecraft System Data, Mission Planning System Data, Flight Dynamics Data, Ground Segment Data, Graphical Products Delivered to CAA in native format (mostly Excel & Word) CAA will be responsible for transcription to pdf and production of metadata. A small sub-set of Excel products to be converted to CAA digital data sets/event lists. Products delivered for 2004 Prime, Summary and JSOC Parameter Products Available as validated, self describing CDF files from CSDS Conversion to CEF possible via Qtran Useful as a quality check for full resolution products CAA Implementation Review 31st May 2005

CAA Implementation Review CAA RDM Delivery For extended mission RDM CD-ROM delivery from ESOC will cease CAA will provide RDM data accessible over the network. DDS access will continue unchanged. Meeting held with ESOC to discuss I/F between ESOC and CAA. Testing confirmed that network delivery, ESOC to CAA, is viable. CD-ROM directory structure will be maintained. Test transfer of 650 MB best case of 4 mins, 2.4 MB/s (uncompressed) Details of CAA to community interface need to be clarified. Delivery of full directory structure (Zip, Rsync etc.). Or user select parts of the RDM directory structure that they want (e.g. by instrument) and just these get delivered. Discussed at IWG earlier this month, document outlining options to be distributed to instrument teams and data centres next week. CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Infrastructure - Status CAA Hardware Status Room at ESTEC has been refurbished (air con etc) System hardware was delivered during August 2004 Basic configuration (operating system and most third party software, CVS, web server etc) is now complete. System is running Debian version of Linux. (not SUSE) Some problems encountered with high reliability component (failover) of this high reliability system! High speed network connection via ESAGrid (1 Gbit/s). Development of CAA data ingestion and cataloguing components. Basic web access capability in place. CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Infrastructure – System/Network System INTERNET s u r f n e t E Q U A N T ESAgrid R ESTEC Front - End network CAA1 CAA2 ESOC RAL 770 www.lpce.cnrs - orleans.fr WHISPER 200 msslx.mssl.ucl.ac.uk PEACE 850 sun2.mpae.gwdg.de RAPID www.cluster.irfu.se EFW 120 cassini.physics.uiowa.edu WBD 95 cis.cesr.fr CIS 250 rhea.iwf.oeaw.ac.at ASPOC 750 mpe.mpg.de EDI Average Speed KB/s Address Data msslhx.mssl.ucl.ac.uk www.cluster.rl.ac.uk 650 cddsa.esoc.ops.esa.int 90 Network CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Infrastructure – Software Development Conceptual design unchanged from System Specification Review Development activities concentrated on data ingestion and cataloguing. Prototype ingestion system in place, tested with more than 20k test and delivered CEF files. Basic Web data access system allows simple queries and retrieval of packaged data. Third party products such as QSAS updated to support CAA products. CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Infrastructure – Web Access Initial login Top level of “Basic” search CAA Implementation Review 31st May 2005

CAA Implementation Review CAA – QSAS support for CEF-2 C4 C2 C3 C1 1 minute CAA Implementation Review 31st May 2005

CAA Implementation Review CAA Future Planning 2005 2006 Reviews ♦ Data S/W Dev. H/W Dev. RDM Deliv. CAA Implementation Review 31st May 2005

CAA Implementation Review Summary After a delayed start-up the CAA activities are making good progress. The CAA standards work provides a firm basis on which to build the high quality science products. A highly productive set of meetings have been held with the instrument teams to define the delivered products. Delivery of Level 1, initial science data and some ancillary products (e.g. plot files) to the CAA is underway. A basic system to allow selection and retrieval of data is in place The details of these activities will be provided in the following presentations. CAA Implementation Review 31st May 2005

CAA Implementation Review Review Presentations Tuesday Afternoon Data Format Working Group Report Tony Allen Data Model and Data Dictionary WG Reoprt Chris Harvey Standards tools – CAA tools Tobias Eriksson 14:15 Coffee Fields and Waves Instrument Status Tobias Eriksson DWP Keith Yearby EFW Yuri Khotyaintsev FGM Jonathan Gloag STAFF Laurent Mirioni WHISPER Bozan Güngör WBD (TBC) End CAA Implementation Review 31st May 2005

CAA Implementation Review Review Presentations Wednesday 08:45 Start Overview of particles status Sinead McCaffrey ASPOC Harald Jeszenszky CIS Alain Barthe EDI Edita Georgescu PEACE Rob Wilson RAPID Stefan Mühlbachler 10:35 Coffee ESOC (TBC) CAA System Infrastructure Steve Esson CAA Software Development Sinead McCaffrey CAA future development plans, next steps Chris Perry 12:00 Summary/Q & A (All) 12:30 End of open session CAA Implementation Review 31st May 2005