1. Cluster Active Archive Overview
Chris Perry and the CAA team
3rd Operations Review
ESTEC
20/21st May 2008

2. CAA 3rd Operations Review
Introduction
Overview
Management
Development
Product Status
User Access
Summary
CAA 3rd Operations Review
20/21st May 2008

3. CAA 3rd Operations Review
Overview
CAA 3rd Operations Review
20/21st May 2008

4. CAA 3rd Operations Review
Introduction
The purpose of this Review is to consider the activities that have been undertaken by the instrument teams and project during the last year.
The instrument teams have given a summary of their activities and status. Here we provide an overall summary of the CAA development and status.
Some of the key milestones during last year include:-
Delivery and ingestion of data to end of first extension ( )
Updates to data ingestion system
Updates to both web and command line request mechanisms for non-CEF
Some changes to layout of web interface to ease selection of science data
Population of the pre-generated CAA plots to end of 2004
Development of on-demand plotting for user intervals
IDL libraries for reading and plotting of CEF products
Progress on improvements in system performance
Ongoing Cross-Calibration and Science User Working group activities
Combined CAA/Cluster workshop
CAA 3rd Operations Review
20/21st May 2008

5. CAA 3rd Operations Review
Introduction
The CAA 2nd Ops Review Package consists of the following documents:
CAA-EST-RP-040 CAA Third Operations Review Executive Summary
CAA-EST-RP-041 Update on recommendations and actions from OR2
CAA-EST-RP-042 Summary of CAA products
CAA-ASP-RP-OR3 ASPOC Annual Report
CAA-CIS-RP-OR3 CIS Annual Report
CAA-DWP-RP-OR3 DWP Annual Report
CAA-EDI-RP-OR3 EDI Annual Report
CAA-EFW-RP-OR3 EFW Annual Report
CAA-FGM-RP-OR3 FGM Annual Report
CAA-PEA-RP-OR3 PEACE Annual Report
CAA-RAP-RP-OR3 RAPID Annual Report
CAA-STA-RP-OR3 STAFF Annual Report
CAA-WBD-RP-OR3 WideBanD Annual Report
CAA-WHI-RP-OR3 WHISPER Annual Report
In addition a set of reference documents was also provided, including a couple of tech notes on new CAA features and updated CEF and MDD.
CAA 3rd Operations Review
20/21st May 2008

6. CAA 3rd Operations Review
Management
CAA 3rd Operations Review
20/21st May 2008

7. CAA 3rd Operations Review
CAA Structure – Team
CAA project is managed by ESA RSSD at ESTEC
Development of CAA draws on expertise from within ESA, the instrument teams and other groups.
The ESA project team is headed by project manager Harri Laakso.
Technical manager, Chris Perry.
Developers:
Graphical products/Processing, Delphine Hermenet.
S/W Development, DB, web, Sinead McCaffrey.
Steve Esson departed the project in March 08
System management by RSSD Unix support group.
New sys-admin (Richard Turner) started May 08
Science support, Matt Taylor, Philippe Escoubet.
New YGT will start in August 08 will focus on product QA
For the extended mission some reduction in staffing
Please use for initial queries
Greater use being made of ticketing system to track comments
CAA 3rd Operations Review
20/21st May 2008

8. CAA 3rd Operations Review
CAA Structure – Team
CAA team organisation (Spring 08):
CAA 3rd Operations Review
20/21st May 2008

9. CAA 3rd Operations Review
Management
CAA review board asked that Double Star be considered within the context of the ongoing CAA activities
No additional funding for this activity – best efforts (must not impact CAA)
Most instrument teams have submitted initial ICDs
Arnaud Masson has done an initial assessment
Some recommendations to improve consistency (e.g. naming)
Technical working and science working groups
Activities continuing at a low level (new versions of CEF spec and MDD)
Cross calibration working group has been active (see next presentation)
Two cross-calibration WG meetings held
Minutes of meetings and presentations on the CAA web site
Continues to be a vital forum to address data quality issues
Cross Calibration
Meeting
Actions
OPEN
CLOSED
NEW
1st Feb 2006 - 1
2nd May 2006 3
3rd Oct 2006 12
4th Feb 2007 4 8 7
5th May 2007 14
6th Oct 2007 10 9
7th Mar 2008 6
CAA 3rd Operations Review
20/21st May 2008

10. CAA 3rd Operations Review
Management
Joint Cluster Workshop and CAA school held March 2008
Brought together researchers, archive developers and tool providers
Combined programme of tutorials, science talks, technical and hands-on
About 120 participants
Proceedings to be published by Springer
Other CAA communication activities
Posters and talks at conferences (AGU and EGS)
Participation in NASA/SPASE activities
Support for tool developers and interoperability initiatives
CAA 3rd Operations Review
20/21st May 2008

11. CAA 3rd Operations Review
CAA Development
CAA 3rd Operations Review
20/21st May 2008

12. CAA 3rd Operations Review
Standards
The CAA standards form the basis of a consistent approach to the formatting and description of the digital data products being handled by the CAA.
The major standards development work was undertaken during the implementation phase by the data format and meatadata working groups.
The Cluster Exchange Format (DS-QMW-TN-0003), has proved very stable.
Minor update to address incompatibilities between spec and MDD
Record count now included in comment at end of file.
The data model and data dictionary (CAA-CDPP-TN-0002) define the semantics used to describe the data.
The data dictionary has also proved stable although it is continuing to be developed and enhanced in support of the instrument team product deliveries.
COORDINATE_SYSTEM = “Instrument”
(MIN/MAX)TIME_RESOLUTION inconsistency.
Multi-instrument/spacecraft product support.
CAA 3rd Operations Review
20/21st May 2008

13. CAA 3rd Operations Review
Support Tools
Matlab CEF toolbox
Provides Matlab interface to the CAAtools software library.
Allows the reading, writing and manipulation of CEF formatted data files and metadata from within Matlab.
Open issue: Please ensure that TZ environment variable is not set
An extension has been provided by Arnaud Masson that provides improved accessibility when accessing a sub-set of the CAA products.
Documentation and source files provided on CAA web site to allow extension to other products.
CAA 3rd Operations Review
20/21st May 2008

14. CAA 3rd Operations Review
Support Tools
IDL CEF_READ software
Generic IDL interface to read CAA CEF files
Options provided to sub-set file by parameters or by time interval
Time converted to MJD or CDFepoch for easy handling within IDL
Simple syntax (see below)
Basic plotting examples provided
See CAA-EST-TN for details
{result} = cef_read([ {filename} , ]
[ /NODATA , ][VAR={names} , ]
[ SEARCHTAG={attr_name} , ]
[ /NODEPEND , ][ /NOCONV , ]
[ /JULDAY , ][ /CDFEPOCH , ]
[ /TRANGE={time_range} ])
CAA 3rd Operations Review
20/21st May 2008

15. CAA 3rd Operations Review
Support Tools
IDL PLOT_CEF software
Generic IDL interface to plot CEF files (uses CEF_READ)
Handles scalars, vectors and multi-dimensional arrays
Based on software used for CAA on-demand plotting
Many options provided to configure plots, labels colour scales etc.
Simple syntax (see below)
See IDL-CEF-PLOT technical note, provided in review package, for details
PLOT_CEF, filename, [VAR_LIST=names,][TRANGE=time,]
[YRANGE=Yrange,] [ZRANGE=Yrange,] [YLOG=Ylog,]
[ZLOG=Zlog,] [SCALE_FACT=scale,] [XCHARSIZE=xcharsize,]
[THICK=thickness,] [CHARSIZE=charsize,] [VAR_TITLE=vtit,]
[PLOT_TITLE=ptit,] [Y_TITLE=ytit,] [Z_TITLE=ztit,]
[VAR_TIT_TXT=vtit,][Y_TIT_TXT=ytit_tab,] [Z_TIT_TXT=ztit_tab,]
[PLOT_TIT_TXT=ptit,][NUMPT_X=num_points,]
[NUMPT_Y=num_points,][PS=ps_file,][/MULTI_PS,][/SINGLE_PANEL]
CAA 3rd Operations Review
20/21st May 2008

16. CAA 3rd Operations Review
Support Tools
CAA 3rd Operations Review
20/21st May 2008

17. CAA 3rd Operations Review
Support Tools
CAA 3rd Operations Review
20/21st May 2008

18. CAA 3rd Operations Review
CAA Infrastructure – System/Network
Data storage and backup
Fibre switch upgrade to support new, faster, RAID device.
Additional 15TB of storage has been ordered.
Some old storage no longer on maintenance (used for scratch space).
Data processing system
Reliability problems with some of the existing hardware.
Investigating UPS to provide graceful shutdown on power failure.
Replacement web server and database server machines (caa5 & caa6).
New 8 core processing system for pipeline processing.
Additional 8 core DB server to improve ingestion performance.
Network connectivity
Network access to public internet provided via ESAgrid.
No major problems with connectivity.
CAA 3rd Operations Review
20/21st May 2008

19. CAA 3rd Operations Review
CAA Infrastructure – Software
The CAA is based around a database and data management system that provides a robust and extensible framework for handling the data products delivered to the CAA.
The main software development activities have concentrated on:
Database replication between the catalogue handler and web server systems
Automation of ingestion system and support for parallel processing
Update to the ingestion, web and command line interfaces for delivery non-CEF products
Improvements to web layout.
Development of IDL packages for reading and plotting CEF data.
Extension of the pre-generated graphical plot coverage.
Provision of an on-demand option for the CAA graphical products.
Additional web services such as workshop support pages, user information update and mail list registration.
CAA 3rd Operations Review
20/21st May 2008

20. CAA 3rd Operations Review
CAA Infrastructure – Software
Web interface product groupings
CAA 3rd Operations Review
20/21st May 2008

21. CAA 3rd Operations Review
CAA Infrastructure – Software
Web user details update interface
Mail group registration interface
CAA 3rd Operations Review
20/21st May 2008

22. CAA 3rd Operations Review
CAA Infrastructure – Software
Pre-generated plots
On-demand plots
CAA 3rd Operations Review
20/21st May 2008

23. CAA 3rd Operations Review
Product Development
CAA 3rd Operations Review
20/21st May 2008

24. CAA 3rd Operations Review
CAA – Instrument Team Delivery Status
Team
OR2 (April-07)
OR3 (May-08)
ASPOC / /
CIS / /
DWP / /
EDI / /
EFW / /
FGM / /
PEACE /
RAPID
STAFF / /
WHISPER /
WBD
ESOC
CAA 3rd Operations Review
20/21st May 2008

25. CAA 3rd Operations Review
CAA – Instrument Team Delivery Status
The total number of active CEF and other format data files that have been ingested into the CAA catalogue is in excess of 3.5 million
The total volume uncompressed volume equivalent to over 40 TB.
Pre-generated plot panels amount to more than twice this number.
In addition there remains a large volume of raw and some Level-1 data that is not yet available via the CAA catalogue system but that are being added as time and resources permit.
CAA 3rd Operations Review
20/21st May 2008

26. CAA 3rd Operations Review
CAA Operations
CAA 3rd Operations Review
20/21st May 2008

27. CAA 3rd Operations Review
CAA – User Access
Number of CAA Users now stands at about 650 (up 250 over last year)
Number of User sessions currently about 500 per month
CAA 3rd Operations Review
20/21st May 2008

28. CAA 3rd Operations Review
CAA – User Access
GB/month
Data volume downloaded per month has increased from ~10 GB per month at OR2 to few 100s GB now.
CAA 3rd Operations Review
20/21st May 2008

29. CAA 3rd Operations Review
CAA – User Access
Breakdown of data downloads by instrument team and month
CAA 3rd Operations Review
20/21st May 2008

30. CAA 3rd Operations Review
Summary
CAA 3rd Operations Review
20/21st May 2008

31. CAA 3rd Operations Review
Summary
The CAA has made significant progress in the past year:
CAA has data covering to for all instruments
Have received more than 3.5 million files
Equivalent uncompressed size of holdings is more than 40TB
About 650 users (~500 user sessions per month)
Data downloads of ~100GB month
Work has continued on the development of the core functionality of the system including:
Completed implementation of support for non-CEF products.
Improvements to product selection pages.
Various other web page updates and improvements (e.g. user details)
Added parallel processing support for ingestion component
Pre-generated and on-demand plotting systems
Production of a IDL toolbox for reading and plotting CAA CEF data.
Progress on improvements in system performance (DB replication).
Ongoing Cross-Calibration and Science User Working group activities.
Joint Cluster Workshop and CAA school
CAA 3rd Operations Review
20/21st May 2008