1. Page 1 Envisat Validation Workshop, Campaign Database, 12/12/2002 Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgroup Campaign Database Terje.Krognes, tk@nilu.notk@nilu.no Aasmund.Fahre.Vik, afv@nilu.no afv@nilu.no Norwegian Institute for Air Research (NILU)

2. Page 2 Envisat Validation Workshop, Campaign Database, 12/12/2002 Presentation Outline Data Centre Status Tools  Local end-user tools  WWW search tools Commissioning Phase Activities

3. Page 3 Envisat Validation Workshop, Campaign Database, 12/12/2002 Data flow Signed protocol required for access to tools, docs and files ECMWF data requires additional protocol ASC2HDF SEARCH HDF2ASC OK? DATA DATABASE table.dat DOCS DATA ORIGINATOR DATA USER FILES nadir web ftp to zardoz http Error email http or ftp

4. Page 4 Envisat Validation Workshop, Campaign Database, 12/12/2002 Data Centre Status (1/3): Hardware The machines have been swapped between production and spare servers The hardware has not been changed Firewall zardoz ftp, telnet http Storage Internet spare nadir

5. Page 5 Envisat Validation Workshop, Campaign Database, 12/12/2002 Data Centre Status (2/3): Software Web-server (nadir) runs Linux Redhat 7.2 with up2date ~weekly (6.2 at 2 nd rehearsal) File server (zardoz) runs Solaris 7, as before Nusphere MySQL  Apache 1.3.22 (1.3.14)  MySQL 3.23.44 (3.23.32)  PHP 4.0.6 (4.0.4pl1) IDL 5.4, as before HDF 4.1r3, as before No re-boot on nadir since April 2002 Uptime on zardoz since April 2002 is approximately 99,98% (2 half-hour maintenance + re-boot)

6. Page 6 Envisat Validation Workshop, Campaign Database, 12/12/2002 Data Centre Status (3/3): Metadata Metadata Guidelines updated by metadata board on 24 April 2002 During commissioning phase, updates of various metadata table entries, without affecting structure:  Project-related information  Actual descriptors Guidelines still applicable, but more clarifications to be added.

7. Page 7 Envisat Validation Workshop, Campaign Database, 12/12/2002 New user tools Major re-write of the ASC2HDF file generation and checking tool around launch 6 smaller upgrades of the ASC2HDF package after launch Entire ASC2HDF source code (FORTRAN) published to users. HDF output routines may be combined with user’s own input and processing modules IDL-tool to generate HDF data files (without error checking): IDL2HDF IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)

8. Page 8 Envisat Validation Workshop, Campaign Database, 12/12/2002 New web functionality - ECMWF ECMWF plots (NH 2,5º*2,5º and T106) ECMWF 10-day back trajectory plots T106 extractions to HDF files

9. Page 9 Envisat Validation Workshop, Campaign Database, 12/12/2002 New web functionality - search and plot Use of defaults Facility to save and retrieve a search Provision for downloading multiple files in tarball via web interface Improved on-line plotting (also for multidimensional data) Corrected display of comments to variables

10. Page 10 Envisat Validation Workshop, Campaign Database, 12/12/2002 File searching (1/4) Defaults are marked with *** Page initially opens without defaults Page may be reset or set to saved defaults bys selecting a previously saved query Files may be filtered by a large number of metadata parameters

11. Page 11 Envisat Validation Workshop, Campaign Database, 12/12/2002 File searching (2/4) The default criteria page lists all criteria sets that this user has previously saved Entries may easily be removed The main search page will be loaded with the selected defaults when launched from this page Defaults may include 4D space- time parameters and variable name wild-cards

12. Page 12 Envisat Validation Workshop, Campaign Database, 12/12/2002 File searching (3/4) The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists. Variable name and variable mode accept multiple selections 4D space-time criteria may be entered as a single point, or as a max/min box All files that include or touch the 4D criteria are listed as matches

13. Page 13 Envisat Validation Workshop, Campaign Database, 12/12/2002 File searching (4/4) Files may be selected by posting date Files updated after a given date may be selected The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID If no criteria are selected, the entire file list will be requested

14. Page 14 Envisat Validation Workshop, Campaign Database, 12/12/2002 File List The file list displays the current selection criteria A single file may be downloaded immediately Files may be marked and downloaded in a compressed tarball Search criteria may be saved for later re-use (type a meaningful description before saving) The Variables link brings up a variable list and other documentation contained in the file

15. Page 15 Envisat Validation Workshop, Campaign Database, 12/12/2002 Variable List The variable list page displays all parameters contained in a file Variable name, mode and descriptor are displayed The measurement unit and corresponding SI unit are displayed All documentation fields attached to the file are displayed Additional documentation fields attached to a single variable may be displayed through the Comments link Variables may be selected to generate an on-line plot (next slide) Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots

16. Page 16 Envisat Validation Workshop, Campaign Database, 12/12/2002 Variable plotting (on-line browse)

17. Page 17 Envisat Validation Workshop, Campaign Database, 12/12/2002 Administrative tools Web tools to report uploads/downloads Web tools to archive/retrieve outdated data files Web tools to check and maintain index database consistency and integrity

18. Page 18 Envisat Validation Workshop, Campaign Database, 12/12/2002 User activities Users have sent approximately 200 emails with requests for accounts or updates to metadata, or with questions to HDF file generation After launch focus has slowly shifted to requests for help with HDF file formatting issues and removal of outdated files From May to November 2002 users have submitted 5880 HDF data files (570 of these were rejected due to formatting problems, and 238 have later been archived). From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included) There are currently 4741 ACVT files open to users (22 active AOIDs) There are currently 462 MAVT files open to users (5 active AOIDs)

19. Page 19 Envisat Validation Workshop, Campaign Database, 12/12/2002 User activities: Up and Downloads The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz

20. Page 20 Envisat Validation Workshop, Campaign Database, 12/12/2002 Nadir-team activities 300+ interactions with users (of which 50% during Commissioning Phase) 250+ interactions with ESA and the Metadata Board (10% during Commissioning Phase) One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs Maintained and improved shell scripts for automatic file checking and storage Created additional IDL and FORTRAN tools Performed extensive web program development Performed user administration updates Added metadata entries and updated the table.dat file 23 times since launch Developed and maintained scripts and data flow to provide pre-processed ECMWF data extractions and plots on the web Monitored the automatic file reception system (now containing nearly 5000 files totalling 275 MB + index database 20 MB)

21. Page 21 Envisat Validation Workshop, Campaign Database, 12/12/2002 Metadata issues Although many quality checks are performed on the data upon submission to the database: The metadata definition is currently not complete enough to perform automatic checks of the validity of reported measurement units Some components and units may still be inconsistently named or mis-used Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.