1 T.C. TURKISH STATE METEOROLOGİCAL SERVICE DEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES TSMS ARCHIVING SYSTEM Mustafa Sert October 2011

2 OUTLINE  Data Flow  Hardware  Software  Services

3 206 awos stations MSSQL DB SYBASE ASE DB 150 awos stations ORACLE DB 199 awos Stations (Building) ORACLE DB (building) SYBASE IQ DW Data Flow MSS Server GTS Data Model Data Radar / Satallite Data Telecomm. Section NWP Section Remote Sensing Section IT Section Preproc/ MARS/ HSM Servers TUMAS WEB Portal Research Section Model Data Observation Messages (SYNOP,METAR..)

4 Data Inventory Climatological data (1926-Today) Ravinsonde data (1971-Today). Synoptic data(1980-Today) AWOS data (2003 -Today) Aerodrome AWOS data (2003-Today) Metar data (2000-Today) Taf,Sigmet,Airmet data (2008-Today) Open Screen Observation Data (1999-Today) Sea Surface temperature data(Beginning– Today). Inversion analyse data (2006-Today) Ozon/UV Data (2006/1997-Today) Radar Data (2008 – Today) Satallite Data (2008 – Today) Model Data (2006 – Today)

5 Data Inventory Data TypeArchieve Period Archieve - End MediaDaily avg. Obs/Forecast/Data Count Archieved Daily Obs/Forecast/Data Amount Sum of Archieved Obs/Forecast/Data Amount Data Format Synoptic Obs.1980TodayDatabase8800 row/day--151,5 GBDB TodayMARS170,000 record33 MB134 GBBUFR Climate Obs TodayDatabase-- 37,9 GBDB AWOS Obs. (Min.) TodayDatabase205 station/day-- - DB SHIP Obs Today MARS MB20 GBBUFR METAR Obs Today MARS MB21,7 GBBUFR 2000TodayDatabase1700 row/day-- DB TAF TodayDatabase350 row/day-- DB SIGMET/AIRMET/ GAMET TodayDatabase20 row/day-- DB Ravinsonde (TEMP) Obs TodayMARS1550 record1.6 MB8,9 GBBUFR 1970TodayDatabase45000 row/day GBDB Inversion Analyse TodayDatabase468 row/day-- DB Inversion Forecast TodayDatabase2457 row/day-- DB Ozon (Ozonsonde) TodayDatabase 1700 row/mnth-- DB

6 Data Inventory Data TypeArchieve Period Archieve - End MediaDaily avg. Obs/Forecast/Data Count Archieved Daily Obs/Forecast/Data Amount Sum of Archieved Obs/Forecast/Data Amount Data Format Ozon (Brewer) Database1 row/day-- DB UVA-UVB Database row/day-- DB UVA-UVB (Brewer) Database8 row/day-- DB UvIndex (Brewer) Database8 row/day-- DB Prec.Water Analyse Database40 row/mnth-- DB Wind Energy Foracast Data TodayFileSystem90 file200 MB/day7 GBAscii-Text Aerodrome AWOS data TodayFileSystem64 station/month10 MB/month69,1 GBAscii-Text Radar (Ank-Ist-Blk-Zng) Radar (Izm-Mgl) Radar (Ant – Hty) Today MARS-TSM TSM (Tape) (Total)9,0 GB 5792 GB (MARS) 2336 GB (TSM) Binary Uydu - MSG TodayTSM (Teyp) 13,0008,2 GB1585 GB (MARS) 6568 GB(TSM) Binary Uydu NOA TodayTSM (Teyp) 2,2001,8 GB22,9 GB (MARS) 1509 GB (TSM) Binary Uydu SAF TodayTSM (Teyp) 9000,5 GB39,3 GB (MARS) 794 GB (TSM) Binary

7 Data TypeArchieve Period Archieve - End MediaDaily avg. Obs/Forecast/Data Count Archieved Daily Obs/Forecast/Data Amount Sum of Archieved Obs/Forecast/D ata Amount Data Format Model – METU TodayTSM2 File/day23,4 MB/day16,5 GBASCII TodayTSM1 File/day431 MB/day10,8 GBNETCDF Model – MM5-MD TodayTSM-Tape1 File/day2,1 GB/day2362 GBCrayBin Model – MM5-MD TodayTSM-Tape3 File/day1592 GBCrayBin Model – MM5-MD TodayTSM-Tape3 File/dayCrayBin Model – MM5-MD TodayTSM-Tape3 File/dayCrayBin Model – MM5- MD TodayTSM-Tape3 File/dayCrayBin Model – ALARO TodayTSM-Tape72 File/day 2,88 GB/day.fa Model – ARPAGE TodayTSM-Tape4 File/day 1,46 GB/dayNetCDF Model – Image TodayTSM-Tape 1 Tar File/Day (18000 file) 2,64 GB/daypng Model - PrecisTSM-Tape 1,077,730 file-6,77 TBspec Model – WRF TSM-Tape 11,2 GB450 GBtar Model – Dust (Dream8b) TodayTSM-Tape1520 MB135 GBtar.gz Data Inventory

8 Hardware  1 DB Server (HP DL380 G4, Intel Xeon 2 CPU 3.4 Ghz, 6 GB RAM, 2 * 146 GB Internal, 1.8 TB SATA- 1.3 TB SCSI External Disk)  1 DW Server (HP ML370 G4 2*3.0 Ghz, 10 GB RAM, 2 * 146GB Internal, 1,2 TB SATA External Disk)  1 DPP Server (IBM x * Dual core Intel Xeon 3.0 Ghz, 24 GB RAM, 2*146 GB Internal, 3 TB FC External Disk)  1 MARS Server (IBM x * Dual Core AMD 2.80 Ghz, 24 GB RAM, 2*146 GB Internal, 2.5 TB FC, 10 TB SATA External Disk)  1 WebPortal Server (IBM x * Dual core Intel Xeon 3.0Ghz Processor, 24 GB RAM, 2*146 GB Internal, 0.5 TB FC External Disk)  1 HSM (TSM) Server (IBM p series P * CPU 1.5 Ghz Power 5 Processor, 32 GB RAM, 2*146 GB Internal, 1 TB FC External Disk)

9 Hardware  2 IBM N5500 Disk System (2 * 11 TB FC, 2*11 TB SATA Disk, 2 SAN Switch)  1 IBM TS3500 Enterprise Tape Library ( 12 x IBM GB Teyp Drive, Total 290 TB Tape Media (415 Data Cartridge x 700 GB)

10 Hardware TÜMAS System Disk System 11 TB FC 11 Terabyte SATA Teyp Library 290 Terabyte MARS Server Archive/retrieve data on/from MARS or retrieve metadata Web Portal Server TUMAS Web Portal presents the data from databases and Archive System Hierarchical Storage Mgmt. Server (TSM) Manages the data on disk and tape systems. Storage Area Network DPP Server Source Data Data Acquire from different sources and pre-process, (format conversion) Sybase IQ ServerSybase ASE Server

11 Software o Operating System ; Generally Suse Linux 9 / 10, except TSM Servers (AIX 5.4) o Database ; Sybase ASE, Sybase IQ, Empress DB and mysql (in prprocessing server) o Hierarchical Storage Management Software; IBM Tivoli Storage Manager 5.5 o Web Application Software; IBM WebSphere Portal 6.1 o Spesific Software ; MARS ( The Meteorological Meteorological Archival and Retrieval System, from ECMWF) o Spesific Software ; EmosLib, Preprocessing software, Metview/magics, GribAPI from ECMWF

12 Software MARS Meteorological Archival and Retrieval System Manages meteorological fields –Operational data –Research data –Special projects Not a file system –Users are not aware of the location of the data An archive, not a database –Metadata online –Data offline Model Data ; GRIB Observations; BUFR

13 Software MARS Components Client/Server architecture Clients: workstations, supercomputers Servers: supercomputers, dedicated servers Several databases Tape library

14 Software MARS at ECMWF 12.3 PetaBytes of data in ~5 million files, for more than 55 · 109meteorological fields ~ 200 GigaBytes of metadata 50 · 106 fields added daily (14 Terabytes) 500 active users/day executing ~ requests/day ~ 19 Terabytes retrieved daily Analysis, forecast and observations since 1957 (ERA-40) Operational forecast since 1985

15 Software MARS A meteorological language Retrieve, date= /to/ , parameter=temperature/geopotential, type=forecast, step=12/to/240/by/12, levels=1000/850/500/200, grid=2/2, area=-10/20/10/0

16 Software MARS What is a field? Class(Operations) Type(Forecast) Stream(Daily Archive) Version(1) Date( ) Base Time(12 UTC) Step(240 hours) Parameter(Temperature) Level(1000 hPa)

17 Software MARS Architecture Meta-Data Request Server MARS Data Server File Store RequestReferences Data ReferencesFiles Data

18 Services TUMAS (Turkey Meteorological Archiving System) Web – based application developped by Java. Users can access data online from TUMAS Internal / External users should be a TUMAS member for retrieve data. External users from non-governmental organizations should pay for data. Data size for database ~ 1 TB Data size for TSM (Disk - Tape Library) ~ 70 TB

19 The data presented in can be grouped into 5 titleswww.tumas.dmi.gov.tr Data Order Database (Synop, Radiosonde, AWOS, Metar) Meteorological Data Archiving System (MARS/TSM) ChartsBulletinBooks Turkish Meteorological Data Archiving System TUMAS Services

20 ALADIN Model Data Meteorological Database Synop Climate Metar AWOS Radiosonde Turkish Meteorological Data Archiving System TUMAS Services

21 Services TUMAS has been designed to meet the increasing demand of storing and presenting of meteorological data (observations, radar, satellite, NWP, etc.) TUMAS is the main repository of meteorological data at Turkish State Meteorology Service (TSMS). It contains terabytes of data which are generated by TSMS. TUMAS use Sybase RDBMS and MARS software which is used in ECMWF for achieve and retrieve data which are stored in different media ( disk storage system,tape library ). MARS software is installed in TUMAS and mainly works archieve/retrieve data from/to disks or tapes. Turkish Meteorological Data Archiving System TUMAS

22 Services In TUMAS – mars section, all data is archieved in bufr and grib format. Because MARS software does not support data except bufr or grib format. TUMAS archieving data includes: observation data (mars, sybase db) raw and product data obtained from satellite raw and product data obtained from meteorological radars input / output data obtained from local and global weather forecast models. Turkish Meteorological Data Archiving System TUMAS

23 Services TUMAS system consists of 4 modules. These modules are Preprocessing, Mars main system, HSM and WebPortal. 1.Preprocessing module : includes below collection of all data decode data convert data to bufr/grib format data quality control write to database to utilized by MARS 2. MARS module : MARS is a client/server based application software to archieve data into disk storage system and tape systems. MARS has a interface script language to archieve/retrieve data. MARS receives data from preprocesing and sends these data to archieve storage system by using HSM interface. Turkish Meteorological Data Archiving System TUMAS

24 Services 3. HSM module : data are archieved in several media (disk and tape systems). To manage data which are stored in this storage system, HSM (hierarchical Storage Management software is utilized. In TUMAS, IBM Tivoli Storage Management software is used. 4. WebPortal : Portal can access to database data for observations. MARS data is presented to users by webmars module. Webmars is data web-based presentation system. All archieved data is presented under several authorization and authentication mechanisms. In TUMAS, SUSE Linux 9 platform is used for mars,webPortal and preprocessing modules. AIX 5 Unix platform is used for HSM module. Turkish Meteorological Data Archiving System TUMAS

25 Future Plans  Installing a new server for climate models regCM and precis. (8 processor, 10 core /proc, 256 GB RAM, 50 TB SAS disk) (2011 Q4 – 2012 Q1)  Installing a virtualization platform (2012 Q3)  Developping a new web portal and grow mapping, visualisation. (with open source software)

26 Questions? Turkish State Met. Service Department of Research And Information Technologies Information Technology and Statistics Division October – 2011 Alanya