Meteorological Data Unified Service Interface System of CMA

Meteorological Data Unified Service Interface System of CMA
An Introduction to Meteorological Data Unified Service Interface System of CMA ( MUSIC ) Wenchun of CMA 2016/11/21, Guangzhou Good morning Introduce Pleased Applications OK, Let’s begin

Content 1. Why We Need MUSIC 2. What does MUSIC Bring to Us
3. How has MUSIC Developed 4. Where will MUSIC Go Next I’ll describe Firstly, why developed Then, describe benefits, especially good features Then, focus on , discuss how realized At last, look forward to OK, why

Applications Need API to Get Data
SWAN MESIS SWAP/SMART IDATA ASOM Applications MICAPS RAISS CIPAS …… Access difficult, Always adjusted, Difficultly popularized Access database directly Multiple Technology Data Modification New Data New Technology update develop Oracle, MySQL, SQL Server, GPFS, … Cassandra, MongoDB, Redis HDFS, DRDS, OSS, … Data Storage Scattered Data Surface data/Upper-air data Surface Integration Upper air Integration Format Standardization … Radar Satellite NWP Ocean Disaster GIS … FY GRAPES Merged Data … As we know, every application need meteorological data. In tradition, applications always access database directly to get data. But in this mode, there are several disadvantages. Firstly, data access is difficult, application developers should be familiar with the language of databases, such as Oracle, MySQL, etc. [ˌet ˈsetərə] .But in reality, many of our developers, especially for meteorological experts, are not familiar with them. Secondly, applications always need to be adjusted with the change of data storage. Along with the development of IT technology, new data storage technology appears rapidly. Besides, New data and New data formats also appear rapidly. So, applications need to be adjusted with all of the change. At last, because of the applications are bound with data storage system, It’s difficult for them to be popularized. In every province, application should be Adapted to local database.

Applications Need API to Get Data
SWAN MESIS SWAP/SMART IDATA ASOM Applications MICAPS RAISS CIPAS …… API ( Data Fetch / Store ) Access easy, Application stable, Easily popularized Multiple Technology Data Modification New Data New Technology update develop Oracle, MySQL, SQL Server, GPFS, … Cassandra, MongoDB, Redis HDFS, DRDS, OSS, … Data Storage Scattered Data Surface data/Upper-air data Surface Integration Upper air Integration Format Standardization … Radar Satellite NWP Ocean Disaster GIS … FY GRAPES Merged Data … So, we need a software layer to decouple applications and data storage system. Provide data APIs, including data fetch and data store. In this mode, data access will be easy, and application will be stable and can be easily popularized.

Applications Need API to Share Functions
SWAN MESIS SWAP/SMART IDATA ASOM Applications MICAPS RAISS CIPAS …… Same functions, Develop and use for oneself Different algorithm, Different Data Same period in previous years Area rainfall Contour analysis Profile analysis Station information Surface data visualization Radar data visualization Satellite data visualization NWP data visualization Multiply data analysis Message service GIS service -WMS GIS service -WFS GIS service -WCS Metadata service … Besides meteorological data, applications also need many functions to build the system. In reality, applications have many of the same functions, such as contour analysis, data visualization, GIS services. But they are always developed and used independently. It’s not necessary. We can develop a function and share it as an API to other applications. In this mode, We can increase our efficiency, and make sure the functions and data we used in every application are the same. contour[ˈkɒntʊə(r)] profile[ˈprəʊfaɪl]

Applications Need API to Share Functions
SWAN MESIS SWAP/SMART IDATA ASOM Applications MICAPS RAISS CIPAS …… Develop & Register API Use API API ( public functions ) Same period in previous years Area rainfall Contour analysis Profile analysis Station information Surface data visualization Radar data visualization Satellite data visualization NWP data visualization Multiply data analysis Message service GIS service -WMS GIS service -WFS GIS service -WCS Metadata service … We can develop a function and share it as an API to other applications. In this mode, We can increase our efficiency, and make sure the functions and data we used in every application are the same.

Why We Need MUSIC CMA’s DaaS (Data as a Service)
1. Decouple applications and data storage Data-fetch API, Data-store API 2. Share application functions: Public-function API Cloud Architecture: IaaS, PaaS, SaaS + DaaS MUSIC : Meteorological Data Unified Service Interface System of CMA So, we developed an API system, MUSIC In MUSIC, we achieved 2 targets. Firstly, decouple………, provide APIs for data fetch and data store. Then, we can share…., provide more and more APIs of public function.

3. How has MUSIC Developed 4. Where will MUSIC Go Next OK, take a look at the advantages MUSIC can bring to us.

MUSIC - Brings what? ( to Users & Applications )
Stable Calling Simple Rich Functionality Self-help API Standard Cross-platform, Multi-language, Multi-protocol (for Different Applications) Data-fetch API Online Interactive Experience ( API Calling & Test ) Stable User Calling Data-store API Configure New API Public-function API Inter-access between State and Provinces Detailed API Information ( for Developers ) Analysis of User’s Own Behavior User Behavior Analysis Especially for users and applications. We summarized in 4 aspects. Firstly, It makes application stable. We developed the API standard. Based on the standard, It makes sure the user calling stable. We can configure new API for new application scenarios. Realize API synchronization nationally to implement inter-access between state and provinces. Secondly, MUSIC makes user calling simple For different applications, All of the APIs MUSIC provided are cross-platform, support multiple languages and protocols. Thirdly, MUSIC provides rich functionality. including data-fetch API, data-store API, public-function API. And we can analyze user’s behavior automatically [ˌɔ:təˈmætɪkəlɪ] Finally, MUSIC provides many self-help functions for people to be familiar with APIs, and to use them. All of the APIs’ information are published online. Users can analyze the own behavior, and can experience the APIs’ calling online. application scenarios

3. How has MUSIC Developed 4. Where will MUSIC Go Next Then, I will share this system with you in the following 4 steps 1.Firstly 2.Then 3. 4.At last, look forward to the future of MUSIC.

MUSIC - Architecture ( Current )
Applications Service Standardized Process Access Data This is the architecture of MUSIC. Its data resource is CIMISS, our data storage system. It provide data to our applications. The system of MUSIC consists of three parts: data access layer, interface encapsulation[inˌkæpsjuˈleiʃən] layer, service release layer. Data access layer achieves basic functions, including data fetch, store and calculation, etc. Interface encapsulation layer implements the standardized encapsulation of APIs. Service Release layer publish APIs in multiple modes, including C/S service, web service, RESTful API and script service.

How realized the features ?
Application Stable Calling Simple Rich Functionality Self-help API Standard Cross-platform, Multi-language, Multi-protocol (for Different Applications) Data-fetch API Online Interactive Experience ( API Calling & Test ) Stable User Calling Data-store API Configure New API Public-function API Inter-access between State and Provinces Detailed API Information ( for Developers ) Analysis of User’s Own Behavior User Behavior Analysis Especially for users and applications. We summarized in 4 aspects. Firstly, It makes application stable. We developed the API standard. Based on the standard, It makes sure the user calling stable. We can configure new API for new application scenarios. Realize API synchronization nationally to implement inter-access between state and provinces. Secondly, MUSIC makes user calling simple For different applications, All of the APIs MUSIC provided are cross-platform, support multiple languages and protocols. Thirdly, MUSIC provides rich functionality. including data-fetch API, data-store API, public-function API. And we can analyze user’s behavior automatically [ˌɔ:təˈmætɪkəlɪ] Finally, MUSIC provides many self-help functions for people to be familiar with APIs, and to use them. All of the APIs’ information are published online. Users can analyze the own behavior, and can experience the APIs’ calling online. application scenarios

MUSIC – Application Stable
Based on API Standard 1. API name pattern : {SERVICE FUNCTION} {DATA TYPE} {DATA CONTENT} {MAIN CONDITION} y = f ( x ) Meteorological Data API Specification 3. API return format : 3 formats Point Data Structure Grid Data Structure Normal File Structure 2. API Parameter define: 2.1 Each parameter : parameter name assignment format value type value scope usage API Parameter Class Num e.g. data parameter 1 dataCode element and statistical parameter 10 elements time parameter 25 times spatial parameter 27 staIds data attribute parameter typhoon 3 typhGIds radar 6 radarElevs NWP 8 fcstEle satellite 4 channels product 2 sevpProdCode Cooperate with Guangdong, Sichuan, Chongqing, etc., developed the standard "Meteorological Data API Specification (Draft)”, to make sure the stability of API definition and interactions between user clients and the MUSIC server. To make application stable, before developing MUSIC system, we developed the standard first “Meteorological Data API Specification”. to make sure the stability of API definition and interactions between user clients and the MUSIC server. The API standard consists of 3 parts: API name pattern, API parameter define, API return format. In the API parameter define, for each parameter, we clearly defined the parameter name, …… In the API return format, we define 3 formats to cover all of the meteorological data. point data structure, such as surface observation data, grid data structure, such as numerical weather prediction data, normal file structure, such as radar data, satellite data.

Stable User Calling Lightweight Client SDK : Based on ICE & API Standard ICE: an efficient communication middleware Web service / RESTful API : Based on SOAP/HTTP & API standard The Software Framework of MUSIC On the user side, the lightweight client only passes standard parameters to the server, and receives data results with standard format. The client doesn’t contain any other function, so it can ensure stability of user calling. On the user side , we developed lightweight client. The client only …… Our SDK is based on ICE, which is an Our web service and RESTful API is based on standard protocol SOAP & HTTP.

API Definition ( Traditional )
MUSIC – Application Stable Configurable API for Specific Application Scenarios API Definition ( Traditional ) ① Users String[][] getSurfEleByTime( String dataCode, String elements, String times ) Customized APIs Calling simple Need secondary development for new scenarios General APIs Parameter List (81) Users Select Statistics Grid Cutout … String[][] getStaEle( String dataCode, String elements, String[] paramNames, String[] paramValues ) ② Calling complex Needn’t secondary development for new scenarios MUSIC uses a few general APIs to realize service functions, and provides a lot of customized APIs to users. Each customized API is according to an application scenario, defined as a set of metadata, so it is simple and configurable. In MUSIC, the strategy of API definition is very important. In tradition, there are two strategies to define them. On strategy is providing customized[ˈkʌstəˌmaɪzd] APIs to users. Every API is according to a specific application scenario. API is easy to be called. But when new scenarios appeared, MUSIC need secondary development to meet the new requirements. The other strategy is providing several general APIs to users. Each API is powerful. It should be called with a set of parameters. So MUSIC need not secondary development for new scenarios. But it’s not easy for users to call them. They should be familiar with all of the parameters, and select some of them for the scenarios. In MUSIC, we connect customized API and general API with metadata. In this mode, a customized API is actually a set of metadata, so it is simple and configurable

Configurable API for Specific Application Scenarios API Definition ( Traditional ) General APIs Customized APIs Parameter List (81) Users Select Statistics Grid Cutout … Metadata API Definition ( MUSIC ) Calling simple Configure API metadata for new scenarios ① Users Customized APIs Calling simple Need secondary development for new scenarios ② Users General APIs Parameter List (81) Grid Cutout Select Statistics … Calling complex Needn’t secondary development for new scenarios MUSIC uses a few general APIs to realize service functions, and provides a lot of customized APIs to users. Each customized API is according to an application scenario, defined as a set of metadata, so it is simple and configurable. In MUSIC, the strategy of API definition is very important. In tradition, there are two strategies to define them. On strategy is providing customized[ˈkʌstəˌmaɪzd] APIs to users. Every API is according to a specific application scenario. API is easy to be called. But when new scenarios appeared, MUSIC need secondary development to meet the new requirements. The other strategy is providing several general APIs to users. Each API is powerful. It should be called with a set of parameters. So MUSIC need not secondary development for new scenarios. But it’s not easy for users to call them. They should be familiar with all of the parameters, and select some of them for the scenarios. In MUSIC, we connect customized API and general API with metadata. In this mode, a customized API is actually a set of metadata, so it is simple and configurable

Configure New API On-demand API Manager New Data Register Data Match APIs Data Manager Match Example Code Publish Online New Scenarios Configure API Match Data API User e.g. Fetch statistics of temperature in an area over a period of time. e.g. statTempByTimeRange ( dataCode, area, timeRange ) e.g. Surface observation data hourly e.g. e.g. For new data or application scenarios, APIs can be configured and published quickly, without secondary development. So, For new data or application scenarios, APIs can be configured and published quickly, without secondary development. For new data, we register the data online, match some APIs and example code for it, then publish them. For new scenarios, we configure an customized API, match some data and example code for it, the publish them. After the publishment, users can use them immediately. 17

Inter-access between State and Provinces API Synchronization Inter-access Nationally Beijing …… 3. API Definition Synchronization Metadata Center 2. API Definition Verify 1. API Definition Register State Sichuan API Synchronization : A distributed API synchronization mechanism is established, to guarantee API unified among state and provinces. We established an API synchronization mechanism guarantee [ˌgærənˈti:] We have a metadata center, its also the API center. Managers can verify API definitions on it. And All of the 32 nodes can register new API and synchronize APIs defined by other nodes. Based on the unified APIs, MUSIC can switch service to other provinces when local system is abnormal. And the switch is transparent for users. Based on the unified APIs around the nation, One province’s MUSIC can switch service to other provinces when local system is abnormal. And the switch is transparent for users. When the MUSIC or Data storage of one province is in systemic breakdown, we can switch the service route totally to other provinces. When only some data are abnormal, we can switch the service of the particular data to other provinces.

MUSIC – Calling Simple Support Different Application Modes
for Different Platforms Languages Programming habits Calling frequency & amount MUSIC – Calling Simple ( Support Different Application Modes ) For different applications, running platforms, development languages, and programming habits, MUSIC provided unified APIs to make users calling convenient. For …… So, developers can select SDK, or web service, or RESTful API, or Script to get data. These methods of the service are somewhat different. The SDK is suitable for fetching a batch of data, it is applicable to background processing system, such as numerical weather prediction systems. The web service or RESTful API is platform-independent, language-independent. Programming with it is very conveniently. But the data volume in one call should not be too large. The script tool is suitable for scientific researchers. They can get data by configuring script, without programming.

MUSIC – Calling Simple Detailed API Information Published on Website
API Class API List for Each API : Basic Information Calling Methods Parameters Example Code (Multiple Languages) Example Result Data List can be accessed MUSIC – Calling Simple ( Detailed API Information Published on Website ) API List: List all APIs in CIMISS, and publish detailed information of each API that users concerned about. We developed a website to publish the information of API and all of the data in CIMISS. The API information Contains API class, API list. For each API, describes its …, its…, its …, and the data list can be accessed by it.

MUSIC – Calling Simple Detailed API Information Published on Website
Data Information : Data Class Data List for Each Data : Time Serial Element List Detailed Information for NWP Products Example Data APIs Matched MUSIC – Calling Simple ( Detailed API Information Published on Website ) Data List: List all data in CIMISS, and publish detailed information of each data that users concerned about. The data information contains data class, data list. For each data, describes its …, its…, its …, and the data list can be accessed by it. Numerical weather prediction And all of the APIs matched with the data.

MUSIC – Rich Functionality
Functions provided currently For all 402 types of data in CIMISS, published 135 data-fetch APIs, 8 data-store APIs. Select for station observation data Return structure Return HTML Return JSON Return JSONP Return XML Return TEXT Statistics for station observation data Max, Min, Sum, Avg, Count Range filter of element value, Range filter of statistics value Order Top Analysis for NWP products Decode file in Grib1/2 Cutout regional data Extract time serial Download for files Locate files and get information Download a batch of files Fetch file stream SDK JAVA C/C++ C# Python PHP Fortran Web Service Web service RESTful API Select for station information By station network, station level By area Return detailed information Store for station observation data Store Store or Update Update Delete Support array & serialized string Store for files Store Store of Update Update Delete Script Tool Download data Support time pattern MUSIC – Rich Functionality ( Functions provided currently ) For all 402 types of data in CIMISS, published 135 data-fetch APIs, 8 data-store APIs. Select & Statistics for station observation data ( surface, upper air, …) Select & Decode & Cutout for NWP products ( EC, T639, …) Select & Download for data in file ( Radar, Satellite, …) Store & Update data For all of the 402 types of data in CIMISS, MUSIC published 135 data-fetch APIs, 8 data-store APIs. provides … function for …, such as Besides, it also support data store and data update.

MUSIC – Rich Functionality
User Behavior Analysis Multi-dimension Statistics & Prediction Access Quantity Real time, Hourly, Daily, Monthly, Yearly Performance Query time, Success rate, Detailed info. APIs called Statistics by API, API list(often used), API list(less used) Data accessed Statistics by data, Data list(often used), Data list(less used) Users by Application, by Department MUSIC – Rich Functionality ( User Behavior Analysis ) Analyze user’s behavior in multi-dimension, to know the usage of MUSIC in real-time around the nation or in local province. MUSIC can analyze user’s behavior in multi-dimension. From the analysis, we can know the usage of MUSIC in real-time around the nation or in local province, we can know the service hotspots and the bottlenecks of the system. We can analyze the user’s access quantity, MUSIC’s performance, which APIs are called frequently, which data are used most, and which users are most active.

MUSIC – Self-help Interactive Experience Online USERS : SYSTEM :
1. Select data 2. Select an API matched with the data 3. Assign values to parameters 4. Submit request SYSTEM : 5. Execute request in the background 6. Display data results on the web Self-help ( Interactive Experience Online ) MUSIC provides interactive experience online. On the website, users can experience the API calling. Users first select a data, …, …, then submit the request The system receives the request, execute it …. Then the data results will be displayed on the web.

Detailed access information
MUSIC – Self-help Analysis of User's Own Behavior Access quantity Query time Detailed access information Data usage API usage MUSIC – Self-help ( Analysis of User's Own Behavior ) Each API user can analyze their own action on MUSIC website, to exam whether their calling is normal or not. Users can also .. They can analyze their access quantity, query time, which data they used, which APIs the called. And they can check the detailed calling information.

MUSIC – Applications currently
Deployed : the State and 31 provinces. Applications: 90+, such as MICAPS 4, CIPAS 2, etc. NMC-MICAPS4 NCC-CIPAS2 NMIC-IDATA MOPS Sichuan Zhejiang Chongqing Inner Mongolia MUSIC – Applications currently Deployed in NMIC and all of the 31 provinces. Supported core operation systems, such as MICAPS 4, CIPAS 2, etc. As of Oct. 31st, 2016, provided data for more than 90 applications. MUSIC has been deployed …

3. How has MUSIC Developed 4. Where will MUSIC Go Next : API Standard + Open Platform Then, I will share this system with you in the following 4 steps 1.Firstly 2.Then 3. 4.At last, look forward to the future of MUSIC.

Compulsory Recommendatory
MUSIC – Future To Perfect API Standard API Define API name pattern API parameter define Assignment of enumerated parameter’s value e.g. data code, element code, etc. Return code of API Format of fetched data Format of stored data API Develop Service protocol User authentication & authorization User action logging API Publish API deploy API publish API manage Compulsory Recommendatory MUSIC – To Perfect API Standard To improve API standard, and to make it complete and more open . We will make the standard complete and more open. The API standard will be improved in 3 aspects, API define, develop and publish including some Compulsory [kəmˈpʌlsəri] specifications and Recommendatory [ˌrekə'mendətərɪ] specifications.

MUSIC – Future To Develop MUSIC Open Platform
MUSIC – To Develop MUSIC Open Platform Based on the API standard, develop a MUSIC open platform to support API development, registration, management, and publishment. On this platform, everyone can develop and share APIs, such as data resource API, calculation analysis API, visualization API, message service API, etc. Registration [ˌredʒɪˈstreɪʃn]

Thanks ! 33

Meteorological Data Unified Service Interface System of CMA

Similar presentations

Presentation on theme: "Meteorological Data Unified Service Interface System of CMA"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Meteorological Data Unified Service Interface System of CMA

Similar presentations

Presentation on theme: "Meteorological Data Unified Service Interface System of CMA"— Presentation transcript:

Similar presentations

About project

Feedback