Tasks within WP4 & The EDUCE database - A summary WP1, WP2, WP4 and WP5 tasks Jussi Kaurola, Niilo Siljamo, Antti Arola Ozone and UV-Radiation Research Group Finnish meteorological Institute Contributions: Tim Martin (UG), Brian Gardiner&Peter Kirsch (BAS), Harry Slaper&Peter den Outer (RIVM), Ola Engelsen (NILU), Martin Stanek (CHMI), Ann Webb&Richard Kift (UMIST), Massimo Ferraguto (SSF),...

Tasks within WP4 (deliverables): 4.1. Installation of new hardware, permitting... (FMI) 4.2. Installation of WWW data submission tools (FMI) 4.3. Addition of further Oracle structures for non-spectral and ancillary data (FMI, BAS) 4.4. Provision of web pages to provide search access to the Oracle database and guidance to users (FMI) 4.5. Update and maintain metadata pages for the database, including flag for WMO guidelines (UMIST) 4.6. Testing of and report on all database search and access tools, and publishing of user's guide and manual, with copy made available at web site (UG)

The EDUCE database - A summary WP1, WP2, WP4 and WP5 tasks : WP1 & WP2 recent changes in the database screen shots submission facilities retrieval facilities data integrity hardware issues plans for the future FMI tasklist summary

FMI task in WP1: "RAF for ozone and albedo" Antti Arola "The irradiance ratio of snow-covered to snow-free conditions tells how much snow enhances the irradiance on average. The average albedo RAF in Sodankylä, calculated from 11 years of data, is about In other words, snow enhancement is about 19%" Report 1.4 (11pages): results/deliverables/1.4/wp_1_4.pdf

FMI task in WP2: "2.1An evaluation of the impact of atmospheric variability on the detection of trends in UV" Antti Arola The developed methodology offers a way of separating the effect of a single factor, e.g., ozone, cloudiness, aerosols or albedo, on the short- and long-term variability of UV, using measurements of spectral irradiance Report 2.1 (13 pages): deliverables/2.1/wp2.1_finalreport.pdf Arola A., K. Lakkala, A. Bais, J. Kaurola, C. Meleti, and P. Taalas, "Factors affecting short- and long-term changes of spectral UV irradiance at two European stations", JGR, doi: /2003JD003447, 2003.

Recent changes in the database quality flags: old data flagged "manually" incoming data flagged automatically submission: auxiliary data retrieval of data: flags included in the retrieval of spectra broadband data auxiliary data: daily values (hourly values not yet) better information of recent submissions, enhanced user options for output (CHMI)

Data submission facilities spectral and broadband submission has been in operation since 2001 the submission of auxiliary data was delayed due to problems with (old) software now the submission works also for aux data "short and friendly guides" for spectral and auxiliary data spectral guide should be OK (after several iterations) auxiliary guide was revised two weeks ago planned changes: speeding up of submission process make it work more as a background process

Data retrieval facilities changes made since the last meeting new oracle table: measurement_type ( spectral, broadband, ozone, cloud, dursun, snowcover) broadband data can be retrieved daily auxiliary can be retrieved (currently not hourly data) flags included in headers and can be retrieved separately some fixes of previously reported problems planned changes fix problems with hourly auxiliary data new user requirements for WPI design, give us feedback! spectral retrieval somewhat slow and error-prone (queue problems). A completely new way to make SQL query and consequent data handling is planned

flagging of one spectrum  #056 GREEN  GREEN Cloudy 3.639e e e e e+05  GREEN e  GREEN 1.700e  GREEN 1.010e e-06  GREEN 1.850e  GREEN CLOUDS 2.860e e-01  SOME_VARIATION_SCAN 9.240e+00  9.766e-04 slf00567  /UVROOT/DB/DBroot/no_ntnu_trh/1997/ol752/ gxa 56

Data integrity: duplicates  what is a duplicate?  a measured spectrum which has several copies in the database  several different spectra have the same measurement time  what is an overlapping spectrum?  several different spectra have overlapping measurement time  both types are included in the data retrievals  both types need to be removed

Data integrity: duplicates duplicates caused by errors in submitted data automatic identification: very complex task, not possible with current software manual work is needed current status (August run by the BAS tool): belsk bre bru jytot

spurious zeroes during the flagging spectra with only zero values in irradiance column were recorded spectra found only 93 cases with sza < 90 degrees at_boku_sonnblick 2 at_boku_vienna 1 be_iasb_bru 53 fi_fmi_sod 2 fr_ustl_vda 5 gb_umist_reading 11 it_jrc_ispra 27 no_nilu_andoya 8

Hardware status the current hardware is becoming old (purchased in 2000 with $40 000) disks are almost full, Oracle disk at critical level recent submissions have been very large total disk space 106 GB Oracle data size 56 GB Flexstor data size 23 GB CPU power is adequate, flagging might be lagging a bit in the case of large submissions disk failure on last Thursday, which was handled very well by the FMI IT department

Planned changes some hardware upgrading is needed: poor mans solution to disk problem: new disks: 4 x 150 GB (4x600 euro) to current disk box (2 for data, and 2 for mirror) better solution: new disk array (about 15 keuro) and some new disks. Much more tolerant for errors. best solution: new server (20 keuro), new disk array (15 keuro) and new disks a short break in service, a longer break in uploading of new data to Oracle operating system upgrade in November-December causing 2-3 days break upgrading of e.g. WPI related software (TOMCAT)

Planned changes... currently everything is in ozone2.fmi.fi server which is outside FMI firewall ("grey area") an alternative solution: server with Flexstor and WPI in "grey area" Oracle inside firewall a better integration of EUVDB Oracle database to FMI other Oracle databases. Benefits: security maintenance even performance?

FMI task list  follow the link on "General Information Page"  updated regularly  check the "wish list" if you have ideas for new improvements Some remarks: -campaign data  manual submission by FMI  retrieval from www pages needs planning and work

Summary  personal opinion: now the functionalities of the database are rather good  some FMI milestones were delayed considerably  the whole project time was more or less development period, which certainly has caused headache to some users  remaining work: hourly auxliary data  planned improvements:  smoother submission of large datasets  faster retrieval of spectral data  improved WPI design  our interest is to continue with the database maintenance. Funding to be discussed later...