WMO Workshop on Information Management (Geneva, 2-4 October 2017) The Integrated Meteorological Information Service System in CMA: Progress and Outlook GAO Feng Item 5.6

Outline What’s the CIMISS How to store various data in CIMISS how to provide high-quality data service for climate in CIMISS Data aggregation system Challenges and outlook

What’s the CIMISS CIMISS： To keep the standardization, high quality, integrality and consistency of the meteorological data To provide directly support to CMA operations To provide convenient, fast data service To optimize and qualify the data operational flow Functions: data collecting, processing, storing，disseminating and servicing Scope: which including one national center and 31 province centers to support national and local operations. CIMISS is China Integrated Meteorological Information Service System. To join the observation system and core meteorological operations (weather, climate, public service and integrated observation) To support data and products collecting, processing, storing，disseminating and servicing To enable all the observing data to users with most convenient way

How to store various data in CIMISS The service (application) oriented: The primary design principle of CIMISS’s database system is the service oriented. So we have to reorganize the stored data in multi-dimensions, different forms and levels to favor of user’s data querying. Reorganizing the data into an application oriented structure by: access frequency temporal and spatial characteristics file formats statistical features …

How to store various data in CIMISS The semblable framework at different level: CIMISS has developed National database system and province database system, with semblable framework and attributes, and will convenient for spreading over and connecting. CIMISS National DB Operational Sys. (MICAPS、CIPAS……) Local users can not only call data from province database, but also nation’s. Users (……) CMA HQ Data synchronized Data Access CIMISS Local DB Operational Sys. (MICAPS、CIPAS……) Users (……) Province Operations Users County

How to store various data in CIMISS The merged of the real-time and historical data Merging the storage structure of the real-time observations and the historical data management, unify the characteristic values, unit, scale, data types Merging the data from different observation sources, or in different format, into integrated database Create duplicates to meet different data access requirements (e.g, we made netCDF file duplications to support climate services and climate prediction. )

How to store various data in CIMISS The universal data catalog and identification code Assign the unique ID for each data exist in the data processing flow Data registry and admittance mechanism, strong governance to the data Using different range of ID, to identify the ingested data and products from different source With the code, we can convenient to follow the tracks of one data in whole data lifecycle, from observation to data service.

How to store various data in CIMISS CIMISS support standard format and code forms: WMO TAC, BUFR and GRIB netCDF, HDF ASCII Domestic data format (A variety of data formats not always standardized, and are converted in the processing flow )

How to store various data in CIMISS The metadata standard and management system WMO core profile WIS DAR WIGOS interpretation or description metadata information that enables data values to be interpreted in context. information relevant to data that facilitates end users discovery metadata information that facilitates data discovery, access and retrieval

Data types CCI metadata select create modify delete import export Data types CCI metadata

how to provide high-quality data service Use MUSIC to provide data access Meteorological Data Unified Service Interface System of CMA decouple applications and data storage Data access layer : achieves basic functions, including data fetch, store and calculation, etc. Interface encapsulation layer:  implements the standardized encapsulation of APIs. Service Release layer : publish APIs in multiple modes, including C/S service, web service,  RESTful API and script service.

how to provide high-quality data service MUSIC specification Based on API standard developed the standard "Meteorological Data API Specification (Draft)”, to make sure the stability of API definition and interactions between user clients and the MUSIC server. The API standard consists of 3 parts: API name pattern, API parameter define, API return format.

how to provide high-quality data service MUSIC support different application modes So, developers can select SDK, or web service, or RESTful API, or Script to get data. These methods of the service are somewhat different. The SDK is suitable for fetching a batch of data, it is applicable to background processing system, such as numerical weather prediction systems. The web service or RESTful API is platform-independent, language-independent. Programming with it is very conveniently. But the data volume in one call should not be too large. The script tool is suitable for scientific researchers. They can get data by configuring script, without programming For different applications, running platforms, development languages, and programming habits, MUSIC provided unified APIs to make users calling convenient. And MUSIC can support SDK, Web service and Script Call.

how to provide high-quality data service MUSIC provides rich functionality Functions provided currently For all 402 types of data in CIMISS, published 135 data-fetch APIs, 8 data-store APIs. Select for station observation data Return structure Return HTML Return JSON Return JSONP Return XML Return TEXT Statistics for station observation data Max, Min, Sum, Avg, Count Range filter of element value, Range filter of statistics value Order Top Analysis for NWP products Decode file in Grib1/2 Cutout regional data Extract time serial Download for files Locate files and get information Download a batch of files Fetch file stream Select for station information By station network, station level By area Return detailed information Store for station observation data Store Store or Update Update Delete Support array & serialized string Store for files Store Store of Update Update Delete

how to provide high-quality data service MUSIC is configurable There is a set of powerful general APIs in MUSIC. But it so complex that not easy for users to call. So MUSIC provide customized APIs to user. All the customized API is configurable. For new data or application scenarios , APIs can be configured and published quickly, without secondary development.

how to provide high-quality data service MUSIC Inter-access between one state and 31 provinces API Synchronization Inter-access Nationally A distributed API synchronization mechanism is established, to ensure API unified among one state and 31 provinces. There is a metadata center, its also the API center. Managers can verify API definitions on it. And All of the 32 nodes can register new API and synchronize APIs defined by other nodes. All of the 32 nodes keep the same by the national-provincial synchronization mechanism, and any data request from any place can be achieved and get data in whole database system by MUSIC.

how to provide high-quality data service MUSIC for the State and 31 provinces. Applications: 90+, such as MICAPS 4, CIPAS 2, etc. NMC-MICAPS4 BCC-CIPAS2 NMIC-IDATA MOCC-MOPS Sichuan Zhejiang Chongqing Inner Mongolia

data aggregation system State-31Provinces-2000Counties CIMISS Data collecting Focus 3 points: Data normalization Data automation Data checking Local Met. Bureau, BCC & NMC Investigating From last year, CMA start to do the data collecting in whole China. There are 31 provinces and 3 centers to take part in the investigating.

data aggregation system Fill forms Data-recording system Basic inf./Metadata Manual entering Auto. Bulk-importing On the data normalization, we developed data filling forms and data recording system to match the data specification. All users can use manual entering and automatic bulk-importing to record data.

data aggregation system Data collecting flow: investigate - filling - check - evaluating We also make the data collecting operational flow. First, investigating, then do data filling and checking. And also need evaluate and QC for the collecting data, at last.

data aggregation system Disaster data collecting: 97types，2,200,000records by 3628providers

Challenge and Outlook What is challenge in Big data era? Explosive growth of Met-related Data A scale-out infrastructure for better resource management with Cloud for centralized data processing and servicing Requirement for data management: Big data + Cloud （The coupling of digital forecast Grid with multi-source atlas for targeted warning）

Challenge and Outlook — From Stand-alone Systems to Cloud-based Platform CIMISS is not just one data storing system. Operation : 4-level IT structure  one cloud, connecting national center with provincial nodes CIMSS : data storage  data platform, integrating Data-QC, Data Service and data mining Public cloud: play a more and more important role in data sharing and interaction with external users Changes

Challenge and Outlook SaaS PaaS IaaS — Move forward to Cloud-computing infrastructure Public Services Weather Climate Observations desktop Mobile Apps Data Service Data processing Operationa Interoperability Mobile Services SaaS Datasets production MICAPS Mobile applications Standard data access Interfaces CIPAS Data statistics and analysis Observation data and products ASOM Data Visualization PaaS IaaS New CIMISS not only can supply data access service, but also cloud-computing serviced on-line.

Thank you Merci