Cluster Active Archive Overview Chris Perry and the CAA team 3rd Operations Review ESTEC 20/21st May 2008

CAA 3rd Operations Review Introduction Overview Management Development Product Status User Access Summary CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Overview CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Introduction The purpose of this Review is to consider the activities that have been undertaken by the instrument teams and project during the last year. The instrument teams have given a summary of their activities and status. Here we provide an overall summary of the CAA development and status. Some of the key milestones during last year include:- Delivery and ingestion of data to end of first extension (2005-12-31) Updates to data ingestion system Updates to both web and command line request mechanisms for non-CEF Some changes to layout of web interface to ease selection of science data Population of the pre-generated CAA plots to end of 2004 Development of on-demand plotting for user intervals IDL libraries for reading and plotting of CEF products Progress on improvements in system performance Ongoing Cross-Calibration and Science User Working group activities Combined CAA/Cluster workshop CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Introduction The CAA 2nd Ops Review Package consists of the following documents: CAA-EST-RP-040 CAA Third Operations Review Executive Summary CAA-EST-RP-041 Update on recommendations and actions from OR2 CAA-EST-RP-042 Summary of CAA products CAA-ASP-RP-OR3 ASPOC Annual Report CAA-CIS-RP-OR3 CIS Annual Report CAA-DWP-RP-OR3 DWP Annual Report CAA-EDI-RP-OR3 EDI Annual Report CAA-EFW-RP-OR3 EFW Annual Report CAA-FGM-RP-OR3 FGM Annual Report CAA-PEA-RP-OR3 PEACE Annual Report CAA-RAP-RP-OR3 RAPID Annual Report CAA-STA-RP-OR3 STAFF Annual Report CAA-WBD-RP-OR3 WideBanD Annual Report CAA-WHI-RP-OR3 WHISPER Annual Report In addition a set of reference documents was also provided, including a couple of tech notes on new CAA features and updated CEF and MDD. CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Management CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Structure – Team CAA project is managed by ESA RSSD at ESTEC Development of CAA draws on expertise from within ESA, the instrument teams and other groups. The ESA project team is headed by project manager Harri Laakso. Technical manager, Chris Perry. Developers: Graphical products/Processing, Delphine Hermenet. S/W Development, DB, web, Sinead McCaffrey. Steve Esson departed the project in March 08 System management by RSSD Unix support group. New sys-admin (Richard Turner) started May 08 Science support, Matt Taylor, Philippe Escoubet. New YGT will start in August 08 will focus on product QA For the extended mission some reduction in staffing Please use caateam@rssd.esa.int for initial queries Greater use being made of ticketing system to track comments CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Structure – Team CAA team organisation (Spring 08): CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Management CAA review board asked that Double Star be considered within the context of the ongoing CAA activities No additional funding for this activity – best efforts (must not impact CAA) Most instrument teams have submitted initial ICDs Arnaud Masson has done an initial assessment Some recommendations to improve consistency (e.g. naming) Technical working and science working groups Activities continuing at a low level (new versions of CEF spec and MDD) Cross calibration working group has been active (see next presentation) Two cross-calibration WG meetings held Minutes of meetings and presentations on the CAA web site Continues to be a vital forum to address data quality issues Cross Calibration Meeting Actions OPEN CLOSED NEW 1st Feb 2006 - 1 2nd May 2006 3 3rd Oct 2006 12 4th Feb 2007 4 8 7 5th May 2007 14 6th Oct 2007 10 9 7th Mar 2008 6 CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Management Joint Cluster Workshop and CAA school held March 2008 Brought together researchers, archive developers and tool providers Combined programme of tutorials, science talks, technical and hands-on About 120 participants Proceedings to be published by Springer Other CAA communication activities Posters and talks at conferences (AGU and EGS) Participation in NASA/SPASE activities Support for tool developers and interoperability initiatives CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Development CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Standards The CAA standards form the basis of a consistent approach to the formatting and description of the digital data products being handled by the CAA. The major standards development work was undertaken during the implementation phase by the data format and meatadata working groups. The Cluster Exchange Format (DS-QMW-TN-0003), has proved very stable. Minor update to address incompatibilities between spec and MDD Record count now included in comment at end of file. The data model and data dictionary (CAA-CDPP-TN-0002) define the semantics used to describe the data. The data dictionary has also proved stable although it is continuing to be developed and enhanced in support of the instrument team product deliveries. COORDINATE_SYSTEM = “Instrument” (MIN/MAX)TIME_RESOLUTION inconsistency. Multi-instrument/spacecraft product support. CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Support Tools Matlab CEF toolbox Provides Matlab interface to the CAAtools software library. Allows the reading, writing and manipulation of CEF formatted data files and metadata from within Matlab. Open issue: Please ensure that TZ environment variable is not set An extension has been provided by Arnaud Masson that provides improved accessibility when accessing a sub-set of the CAA products. Documentation and source files provided on CAA web site to allow extension to other products. CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Support Tools IDL CEF_READ software Generic IDL interface to read CAA CEF files Options provided to sub-set file by parameters or by time interval Time converted to MJD or CDFepoch for easy handling within IDL Simple syntax (see below) Basic plotting examples provided See CAA-EST-TN-00016 for details {result} = cef_read([ {filename} , ] [ /NODATA , ][VAR={names} , ] [ SEARCHTAG={attr_name} , ] [ /NODEPEND , ][ /NOCONV , ] [ /JULDAY , ][ /CDFEPOCH , ] [ /TRANGE={time_range} ]) CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Support Tools IDL PLOT_CEF software Generic IDL interface to plot CEF files (uses CEF_READ) Handles scalars, vectors and multi-dimensional arrays Based on software used for CAA on-demand plotting Many options provided to configure plots, labels colour scales etc. Simple syntax (see below) See IDL-CEF-PLOT technical note, provided in review package, for details PLOT_CEF, filename, [VAR_LIST=names,][TRANGE=time,] [YRANGE=Yrange,] [ZRANGE=Yrange,] [YLOG=Ylog,] [ZLOG=Zlog,] [SCALE_FACT=scale,] [XCHARSIZE=xcharsize,] [THICK=thickness,] [CHARSIZE=charsize,] [VAR_TITLE=vtit,] [PLOT_TITLE=ptit,] [Y_TITLE=ytit,] [Z_TITLE=ztit,] [VAR_TIT_TXT=vtit,][Y_TIT_TXT=ytit_tab,] [Z_TIT_TXT=ztit_tab,] [PLOT_TIT_TXT=ptit,][NUMPT_X=num_points,] [NUMPT_Y=num_points,][PS=ps_file,][/MULTI_PS,][/SINGLE_PANEL] CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Support Tools CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Support Tools CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Infrastructure – System/Network Data storage and backup Fibre switch upgrade to support new, faster, RAID device. Additional 15TB of storage has been ordered. Some old storage no longer on maintenance (used for scratch space). Data processing system Reliability problems with some of the existing hardware. Investigating UPS to provide graceful shutdown on power failure. Replacement web server and database server machines (caa5 & caa6). New 8 core processing system for pipeline processing. Additional 8 core DB server to improve ingestion performance. Network connectivity Network access to public internet provided via ESAgrid. No major problems with connectivity. CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Infrastructure – Software The CAA is based around a database and data management system that provides a robust and extensible framework for handling the data products delivered to the CAA. The main software development activities have concentrated on: Database replication between the catalogue handler and web server systems Automation of ingestion system and support for parallel processing Update to the ingestion, web and command line interfaces for delivery non-CEF products Improvements to web layout. Development of IDL packages for reading and plotting CEF data. Extension of the pre-generated graphical plot coverage. Provision of an on-demand option for the CAA graphical products. Additional web services such as workshop support pages, user information update and mail list registration. CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Infrastructure – Software Web interface product groupings CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Infrastructure – Software Web user details update interface Mail group registration interface CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Infrastructure – Software Pre-generated plots On-demand plots CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Product Development CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA – Instrument Team Delivery Status Team OR2 (April-07) OR3 (May-08) ASPOC 2001-01-01/2004-12-31 2001-01-01/2006-01-01 CIS 2001-01-01/2002-12-31 2001-01-01/2005-12-31 DWP 2001-01-02/2004-03-25 2001-01-01/2006-12-31 EDI 2001-02-02/2004-12-31 2001-02-02/2005-12-31 EFW 2001-02-01/2005-12-31 2001-01-30/2006-01-01 FGM 2001-01-30/2004-09-02 2001-01-30/2006-10-01 PEACE 2001-01-01/2003-12-31 RAPID STAFF 2001-01-03/2006-12-31 2001-01-03/2007-10-31 WHISPER 2001-02-02/2004-10-31 WBD ESOC CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA – Instrument Team Delivery Status The total number of active CEF and other format data files that have been ingested into the CAA catalogue is in excess of 3.5 million The total volume uncompressed volume equivalent to over 40 TB. Pre-generated plot panels amount to more than twice this number. In addition there remains a large volume of raw and some Level-1 data that is not yet available via the CAA catalogue system but that are being added as time and resources permit. CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA Operations CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA – User Access Number of CAA Users now stands at about 650 (up 250 over last year) Number of User sessions currently about 500 per month CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA – User Access GB/month Data volume downloaded per month has increased from ~10 GB per month at OR2 to few 100s GB now. CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review CAA – User Access Breakdown of data downloads by instrument team and month CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Summary CAA 3rd Operations Review 20/21st May 2008

CAA 3rd Operations Review Summary The CAA has made significant progress in the past year: CAA has data covering 2001-01-01 to 2005-12-31 for all instruments Have received more than 3.5 million files Equivalent uncompressed size of holdings is more than 40TB About 650 users (~500 user sessions per month) Data downloads of ~100GB month Work has continued on the development of the core functionality of the system including: Completed implementation of support for non-CEF products. Improvements to product selection pages. Various other web page updates and improvements (e.g. user details) Added parallel processing support for ingestion component Pre-generated and on-demand plotting systems Production of a IDL toolbox for reading and plotting CAA CEF data. Progress on improvements in system performance (DB replication). Ongoing Cross-Calibration and Science User Working group activities. Joint Cluster Workshop and CAA school CAA 3rd Operations Review 20/21st May 2008