1. WMO Workshop on Information Management
(Geneva, 2-4 October 2017)
The Integrated Meteorological Information Service System in CMA: Progress and Outlook
GAO Feng
Item 5.6

2. Outline What’s the CIMISS How to store various data in CIMISS
how to provide high-quality data service for climate in CIMISS
Data aggregation system
Challenges and outlook

3. What’s the CIMISS
CIMISS：
To keep the standardization, high quality, integrality and consistency of the meteorological data
To provide directly support to CMA operations
To provide convenient, fast data service
To optimize and qualify the data operational flow
Functions: data collecting, processing, storing，disseminating and servicing
Scope: which including one national center and 31 province centers to support national and local operations.
CIMISS is China Integrated Meteorological Information Service System.
To join the observation system and core meteorological operations (weather, climate, public service and integrated observation)
To support data and products collecting, processing, storing，disseminating and servicing
To enable all the observing data to users with most convenient way

4. How to store various data in CIMISS
The service (application) oriented:
The primary design principle of CIMISS’s database system is the service oriented.
So we have to reorganize the stored data in multi-dimensions, different forms and levels to favor of user’s data querying.
Reorganizing the data into an application oriented structure by:
access frequency
temporal and spatial characteristics
file formats
statistical features …

5. How to store various data in CIMISS
The semblable framework at different level:
CIMISS has developed National database system and province database system, with semblable framework and attributes, and will convenient for spreading over and connecting.
CIMISS
National DB
Operational Sys.
(MICAPS、CIPAS……)
Local users can not only call data from province database, but also nation’s.
Users
(……)
CMA HQ
Data synchronized
Data Access
CIMISS
Local DB
Operational Sys.
(MICAPS、CIPAS……)
Users
(……)
Province
Operations
Users
County

6. How to store various data in CIMISS
The merged of the real-time and historical data
Merging the storage structure of the real-time observations and the historical data management, unify the characteristic values, unit, scale, data types
Merging the data from different observation sources, or in different format, into integrated database
Create duplicates to meet different data access requirements (e.g, we made netCDF file duplications to support climate services and climate prediction. )

7. How to store various data in CIMISS
The universal data catalog and identification code
Assign the unique ID for each data exist in the data processing flow
Data registry and admittance mechanism, strong governance to the data
Using different range of ID, to identify the ingested data and products from different source
With the code, we can convenient to follow the tracks of one data in whole data lifecycle, from observation to data service.

8. How to store various data in CIMISS
CIMISS support standard format and code forms:
WMO TAC, BUFR and GRIB
netCDF, HDF
ASCII
Domestic data format (A variety of data formats not always standardized, and are converted in the processing flow )

9. How to store various data in CIMISS
The metadata standard and management system
WMO core profile
WIS DAR
WIGOS
interpretation or description metadata
information that enables data values to be interpreted in context. information relevant to data that facilitates end users
discovery metadata
information that facilitates data discovery, access and retrieval

10. Data types CCI metadata
select
create
modify
delete
import
export
Data types
CCI
metadata

11. how to provide high-quality data service
Use MUSIC to provide data access
Meteorological Data Unified Service Interface System of CMA
decouple applications and data storage
Data access layer : achieves basic functions, including data fetch, store and calculation, etc. Interface encapsulation layer:  implements the standardized encapsulation of APIs. Service Release layer : publish APIs in multiple modes, including C/S service, web service, 
RESTful API and script service.

12. how to provide high-quality data service
MUSIC specification
Based on API standard
developed the standard "Meteorological Data API Specification (Draft)”, to make sure the stability of API definition and interactions between user clients and the MUSIC server.
The API standard consists of 3 parts: API name pattern, API parameter define, API return format.

13. how to provide high-quality data service
MUSIC support different application modes
So, developers can select SDK, or web service, or RESTful API, or Script to get data.
These methods of the service are somewhat different.
The SDK is suitable for fetching a batch of data, it is applicable to background processing system, such as numerical weather prediction systems.
The web service or RESTful API is platform-independent, language-independent. Programming with it is very conveniently. But the data volume in one call should not be too large.
The script tool is suitable for scientific researchers. They can get data by configuring script, without programming
For different applications, running platforms, development languages, and programming habits, MUSIC provided unified APIs to make users calling convenient.
And MUSIC can support SDK, Web service and Script Call.

14. how to provide high-quality data service
MUSIC provides rich functionality
Functions provided currently
For all 402 types of data in CIMISS, published 135 data-fetch APIs, 8 data-store APIs.
Select for station observation data
Return structure
Return HTML
Return JSON
Return JSONP
Return XML
Return TEXT
Statistics for station observation data
Max, Min, Sum, Avg, Count
Range filter of element value, Range filter of statistics value
Order
Top
Analysis for NWP products
Decode file in Grib1/2
Cutout regional data
Extract time serial
Download for files
Locate files and get information
Download a batch of files
Fetch file stream
Select for station information
By station network, station level
By area
Return detailed information
Store for station observation data
Store
Store or Update
Update
Delete
Support array & serialized string
Store for files
Store
Store of Update
Update
Delete

15. how to provide high-quality data service
MUSIC is configurable
There is a set of powerful general APIs in MUSIC. But it so complex that not easy for users to call.
So MUSIC provide customized APIs to user.
All the customized API is configurable. For new data or application scenarios , APIs can be configured and published quickly, without secondary development.

16. how to provide high-quality data service
MUSIC Inter-access between one state and 31 provinces
API Synchronization
Inter-access Nationally
A distributed API synchronization mechanism is established, to ensure API unified among one state and 31 provinces. There is a metadata center, its also the API center. Managers can verify API definitions on it. And All of the 32 nodes can register new API and synchronize APIs defined by other nodes.
All of the 32 nodes keep the same by the national-provincial synchronization mechanism, and any data request from any place can be achieved and get data in whole database system by MUSIC.

17. how to provide high-quality data service
MUSIC for the State and 31 provinces.
Applications: 90+, such as MICAPS 4, CIPAS 2, etc.
NMC-MICAPS4
BCC-CIPAS2
NMIC-IDATA
MOCC-MOPS
Sichuan
Zhejiang
Chongqing
Inner Mongolia

18. data aggregation system
State-31Provinces-2000Counties
CIMISS
Data collecting
Focus 3 points:
Data normalization
Data automation
Data checking
Local Met. Bureau, BCC & NMC
Investigating
From last year, CMA start to do the data collecting in whole China. There are 31 provinces and 3 centers to take part in the investigating.

19. data aggregation system
Fill forms
Data-recording system
Basic inf./Metadata
Manual entering
Auto. Bulk-importing
On the data normalization, we developed data filling forms and data recording system to match the data specification.
All users can use manual entering and automatic bulk-importing to record data.

20. data aggregation system
Data collecting flow:
investigate - filling - check - evaluating
We also make the data collecting operational flow.
First, investigating, then do data filling and checking. And also need evaluate and QC for the collecting data, at last.

21. data aggregation system
Disaster data collecting: 97types，2,200,000records by 3628providers

22. Challenge and Outlook What is challenge in Big data era?
Explosive growth of Met-related Data
A scale-out infrastructure for better resource management with Cloud for centralized data processing and servicing
Requirement for data management: Big data + Cloud
（The coupling of digital forecast Grid with multi-source atlas for targeted warning）

23. Challenge and Outlook
— From Stand-alone Systems to Cloud-based Platform
CIMISS is not just one data storing system.
Operation : 4-level IT structure  one cloud, connecting national center with provincial nodes
CIMSS : data storage  data platform, integrating Data-QC, Data Service and data mining
Public cloud: play a more and more important role in data sharing and interaction with external users
Changes

24. Challenge and Outlook SaaS PaaS IaaS
— Move forward to Cloud-computing infrastructure
Public Services
Weather
Climate
Observations
desktop
Mobile Apps
Data Service
Data processing
Operationa Interoperability
Mobile Services
SaaS
Datasets production
MICAPS
Mobile applications
Standard data access Interfaces
CIPAS
Data statistics and analysis
Observation data and products
ASOM
Data Visualization
PaaS
IaaS
New CIMISS not only can supply data access service, but also cloud-computing serviced on-line.

25. Thank you
Merci