1 T.C. TURKISH STATE METEOROLOGİCAL SERVICE DEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES METEOROLOGICAL DATA MANAGEMENT Mustafa Sert October 2011

2 OUTLINE  Meteorological data definition  Observation Stage  Telecommunication Stage  Archive and Quality Control Stage  Presentation Stage

3 Meteorological Data Google definition; “ Meteorological facts pertaining to the atmosphere, such as wind, temperature, air density, and other phenomena that affect.” Another definition; “All observed, derived, calculated, visualized data for meteorological aims.“ Meteorological data is of vital importance used in scientific and social areas for a sustainable world. Meteorological data is the very basic input in diagnosing the climate change, global warming, weather forecasting, weather pollution, general atmospheric cycle, decision-make, etc..

4 Stages  Meteorological data production (Observation,Calculation, etc.)  Telecommunication of Met.data  Archive and quality control of Met. data  Presentation of Met. data

5 Stage1; Meteorological data production Meteorological data sources are;  Observations of surface stations (Automatic Weather Observation Systems, Manuel Observation Systems, etc.)  Observations of upper-Air Stations (Ravinsonde, etc.)  Observations of sea Stations (Coastal Stations and ships, buoys)  Meteorological Radars  Meteorological Satallites  Meteorological Models (Forecast model, Climate model, Dust model, etc.)

6 Stage1; Meteorological data production TSMS Observing System;  132 Synoptic Stations  257 Climatological Stations (242 AWOS, 15 Manuel)  63 Aviation Stations (50 AWOS, 13 Manual)  Total 463 AWOS Stations, 246 installing (2012)  40 Sea Stations (installing, 2012)  8 Ravinsonde Stations  8 Meteorological Radars ( 4 installing)

7 Stage1; Meteorological data production TSMS AWOS Observation Network

8 Stage1; Meteorological data production For good-quality Observations;  Select the correct location. Use WMO Standards.  Select the correct instruments. Use WMO Standards.  Specify the data production standards. (For AWOS, do you need the minutely data for pressure or temperature?)  Specify the station limits for quaility control process.  Create metadata database and open it to users.

9 Stage2; Meteorological data Communication It is important to transport meteorological data from station to centre correctly and in time.  Specify which communication systems you use. It is related to Country’s telecommunication infrastructure.  Specify alternative communication systems. At least 2 communication system is offered for backup.

10 Stage2; Meteorological data Communication TSMS Communication Network includes;  110 VSAT Connection  5 Central Satallite Receiving System (2 EumetCast, 1 L-X Band, 2 L Band) and 4 Satallite Receiving System (EumetCast)  340 GPRS Connection  330 ADSL Connection  8 radio-link connection  150 Mb MetroEthernet Internet connection (Ankara - Centre)  10 Mb MetroEthernet Internet connection (İstanbul - Region)  PSTN (Dial-up) connection  Fax connection

11 Stage2; Meteorological data Communication

12 Stage3; Archive and Quality Control Our data archiving strategy is ;  For small data like observations ; To use relational databases with fast disk systems.  For huge data like radar, satallite and model ; To use storage managemant software with slow & big capacity disk - tape systems.

13 Stage3; Archive and Quality Control Other basics of data archiving; Collect all data in one center and one disaster recovery center for easy management Virtualization technology for database servers may be a good solution. Enterprise tape library for huge data is optimal solution.

14 Stage3; Archive and Quality Control What about the model data ? (like climate models) Climate model outputs consist millions of files. Total volume reaches to terabytes. It is very important to access these outputs fast and reliable. Solution ? Data Aging policy !! ( Bring frequently-used, new data to the fast storage and move old, least-used data to the slow – big capacity storage)  Use fast disks for frequently-used climate model outputs.  Use slow&big capacity disks and enterprise tape librarys with storage management software.

15 Stage3; Archive and Quality Control What we do ? Our Harware and Software platform for climate model is ; HP DL580G5 Server (4 x 2.4 Ghz Intel Xeon Processorwith dual core, 8 GB RAM, 420 GB Internal FC Disk, 12 TB External SATA disk) SLES 10 ES Operating System After a period of time; We archive old data in IBM TS3500 Tape Library with IBM Tivoli Storage Manager Software. TS3500 Tape Library has ; 12 x IBM GB Teyp Drive, Total 290 TB Tape Media (415 Data Cartridge x 700 GB)

16 Stage3; Archive and Quality Control Tivoli Storage Manager Software (TSM) provides archiving all kind of data from Model server’s command prompt. To archive model output from command line use ; #dsmc arc “/home/precis/precis_out/iklao000.iklao.d10182.t leave” To retrieve model output from command line use ; #dsmc ret “/home/precis/precis_out/klao000.iklao.d10182.t leave” Tape Library Robotic System

17 Stage3; Archive and Quality Control Data Quality Control; If your input data is bad, your model outputs are bad as well! Observations should pass the data quality control alghoritms for healthy model inputs. In quality control, we use; Range Test, Step Test, Persistence Test, Like Instrument Test and Spatial Tests

18 Stage 4; Presentation Providing data as fast as possible from archiving system to all internal and external users.  To do this, we use Web Portal. (tumas.dmi.gov.tr)  Web portal can access fast Sybase IQ datawarehouse (OLAP based Database) and MARS System for retrieve requested data.  Data output formats for observations from database may be xml, doc, xls,txt,pdf.  Data output formats for observations from MARS may be bufr.

19 Questions? Turkish State Met. Service Department of Research And Information Technologies Information Technology and Statistics Division October – 2011 Alanya