Meteorological Big Data-as-a-Service: SOA based Environment and Methods for Meteorological Big Data Exploration Yaqiang Wang Chengdu University of Information Technology Joint work with members of Joint Research Center for Meteorological Software Engineering, leaded by Prof. Hongping Shu What a honor. Today I have a chance to give a talk here, titled “Meteorological Big Data-as-a-Service: SOA based environment and methods for meteorological big data exploration”. I am WANG Yaqiang, from Chengdu University of Information Technology. This is a joint work with members of joint research center for meteorological software engineering, leaded by professor SHU Hongping.

Explosive growth of amount of meteorological and related data Aspects of manipulating meteorological data: Saving — large volume, real-time, multi-format, and multi-source Accessing and managing — efficient and effective Sharing — safe and flexible Exploiting — easy and adept Illustrating — user-friendly and knowledge-driven New challenges With the explosively growing of amount of meteorological and related data, new challenges of information technologies are brought to the meteorological research, such as saving large volume of real-time and multi-format meteorological data from multiple data sources, efficiently accessing and managing target data item(s) in a huge amount of data, safely sharing high value data in a flexible way, providing an easy developing mode to let domain experts adeptly use and improve sophisticated learning algorithms, and user-friendly showing illustrative experimental results with knowledge-driven approach.

Explosive growth of amount of meteorological and related data Aspects of manipulating meteorological data: Saving Accessing and managing Sharing Exploiting Illustrating New challenges To attempt these challenges, players have taken several types of approaches and constructed lots of tools. For example, Hadoop, Spark, Storm, Docker, etc.

General Cloud Computing Models for Meteorological Big Data Exploration IaaS PaaS SaaS General Cloud Computing Models for Meteorological Big Data Exploration And, recently, when developing tools and platforms, cloud computing models are frequently used. Moreover, in order to fast do meteorological researches and construct applications for meteorological researchers, or achieve a well “market” adaptability for industry, most players would like to simply follow the general cloud computing models for meteorological big data exploration.

General Cloud Computing Models for Meteorological Big Data Exploration IaaS PaaS SaaS General Cloud Computing Models for Meteorological Big Data Exploration In the general model, IaaS, the short name of Infrastructure as a Service, provides capability to consumers to provision processing, storage, networks, and other fundamental computing resources, where the consumers are able to deploy and run arbitrary software, which can include operating systems and applications. However, consumers need not manage or control the underlying cloud infrastructure, but has to control operating systems, storage and deployed applications, and possibly limited control of selecting network components, e.g. host firewalls.

General Cloud Computing Models for Meteorological Big Data Exploration IaaS PaaS SaaS General Cloud Computing Models for Meteorological Big Data Exploration PaaS, the short name of Platform as a Service, provides the capability to the consumers is to deploy consumer created or acquired applications onto the cloud infrastructure by using programming languages, libraries, services, and/or tools supported by the PaaS providers. Consumers also need not manage or control the underlying cloud infrastructure, but have to control the deployed applications and possibly configuration settings for the application-hosting environment.

General Cloud Computing Models for Meteorological Big Data Exploration IaaS PaaS SaaS General Cloud Computing Models for Meteorological Big Data Exploration Software as a Service, SaaS for short, provides services to consumers to use the provider’s applications running on the cloud infrastructure. The applications are accessible from various client devices through either a thin client interface, such as a web browser. And consumers do not manage or control the underlying cloud infrastructure, or even the individual application capabilities, with the possible exception of limited user-specific application configuration settings.

Domain-Specific Extensions of the General Models IaaS Meteorological Data as a Service (MDaaS) PaaS SaaS Meteorological Information as a Service (MInfoaaS) Meteorological Knowledge as a Service (MKaaS) Wisdom in Meteorological Big Data as a Service (WMBDaaS) Domain-Specific Extensions of the General Models In contrast, according to our knowledge in domain informatization, we propose a domain-specific extensions of the general models. Each layer in the general cloud computing system architecture should be carefully tailored for meteorological domain requirements according to domain-specific characteristics of meteorological data content.

Domain-Specific Extensions of the General Models IaaS Meteorological Data as a Service (MDaaS) PaaS SaaS Meteorological Information as a Service (MInfoaaS) Meteorological Knowledge as a Service (MKaaS) Wisdom in Meteorological Big Data as a Service (WMBDaaS) Domain-Specific Extensions of the General Models IaaS, PaaS and SaaS in the general cloud computing system architecture would keep their original functions in the extended model.

Domain-Specific Extensions of the General Models IaaS Meteorological Data as a Service (MDaaS) PaaS SaaS Meteorological Information as a Service (MInfoaaS) Meteorological Knowledge as a Service (MKaaS) Wisdom in Meteorological Big Data as a Service (WMBDaaS) Domain-Specific Extensions of the General Models However, they should further provide services for meteorological data content services on the same layers respectively, and then services in the same layer would provide basic services for their upper layers.

Domain-Specific Extensions of the General Models IaaS Meteorological Data as a Service (MDaaS) PaaS SaaS Meteorological Information as a Service (MInfoaaS) Meteorological Knowledge as a Service (MKaaS) Wisdom in Meteorological Big Data as a Service (WMBDaaS) Domain-Specific Extensions of the General Models Finally, we could construct intelligent applications based on meteorological big data under domain-specific demand restrict.

Meteorological Data-as-a-Service Provides services based on both current and future raw meteorological (big) data Current Data Data-Sharing Service Standardization Service Future Data Data Access Service Data Integration Service Basic Data Processing Service Meteorological Data-as-a-Service According to the requirement of wisdom meteorology proposed by CMA, Meteorological Data as-a-Service, in more detail, provides services based on both current and future raw meteorological data. For current raw meteorological data, the services primarily include the data-sharing service and the data standardization service. For the future data, the services are mainly for accessing and integrating data. Types of basic data processes are the fundamental services for both current and future data.

Meteorological Information-as-a-Service Extracts meteorological information from data according to user demands, and provides “future” information for users Information for User Demands Basic Data Mining Service Basic Machine Learning Service Basic Natural Language Processing Service Basic Information Retrieval Service Data Curation and Management Services Meteorological Information-as-a-Service Meteorological Information-as-a-Service provides services by using the current and future data. Moreover, it would provide current and future information based on basic data mining, machine learning, natural language processing and information retrieval services as well as data curation and management services. Providing current information is the activity of getting meteorological information from diversified data sources, and providing future information means extracting meteorological information from the data that users need.

Meteorological Knowledge-as-a-Service Provides services with respect to existing or will-be-refined explicit knowledge, e.g. meteorological ontologies. Meteorological Domain Knowledge Advanced Data Mining Service Advanced Machine Learning Service Advanced Natural Language Processing Service Advanced Information Retrieval Service Information Management Service Meteorological Knowledge-as-a-Service Meteorological Knowledge-as-a-Service provides services with respect to existing or will-be-refined explicit knowledge, for example, meteorological ontologies. The existing and will-be-refined explicit meteorological knowledge are obtained based on advanced data mining, machine learning, natural language processing, information retrieval, and information management services. Providing existing meteorological knowledge indicates knowledge querying, and providing will-be-refined meteorological knowledge means to construct new meteorological knowledge bases or expand current available knowledge bases.

Wisdom in Meteorological Big Data-as-a-Service Provides various intelligent meteorological big data applications Heterogeneous and Scattered Meteorological Big Data Intelligent Applications Wisdom in Meteorological Big Data-as-a-Service Based on the meteorological data, information, and knowledge services, correct judgments, decisions, and actions could be provided, and various intelligent meteorological big data applications would be constructed based on the complex, heterogeneous and scattered meteorological big data. Right services for the right objects at a right time and context can be provided to the users and public.

Keys of the Domain-Specific Extensions of the General Models There are several keys in the domain-specific extension of the general cloud computing models for meteorological data exploration.

Keys — Standards for Each Layer Provide evidence for managing and constructing data, information and knowledge sources Two basic standards: standards of properties of instances, information, and concepts standards of relations among instances, information, and concepts Two types of standards: static data content standard: definitions of basic data standards of the current period dynamic data content standard: definitions of basic data standard changes over time Make the extended models have flexibility and adaptability Keys — Standards for Each Layer The first is establishing standards for each layer. We conclude that there are two basic standards including standards of instances’, information’s and concepts’ properties, and relations between instances, information, and concepts in every layer. These standards would provide evidence for managing and constructing data, information and knowledge sources. Moreover, these standards can be further divided into two types, i.e. static standard and dynamic standard. They would make the extended models have flexibility and adaptability. Recently, CMA is working on establishing static type of data content standard. We suggest that, if possible, we should quickly focus our attention on establishing dynamic standards.

Keys — Integrations Data Layer Integration Logic Layer Integration User-Interface Integration Keys — Integrations The second key is integrations including data layer integration, logic layer integration, and user-interface integration. Data layer integration indicates raw data integration, such as database integration, file data integration, real-time data integration, and hard-ware and existing information system integration. Logic layer integration means business logic integration. There are two other types of logic layer integration conditions, including interface connection type integration and intrusion type integration, which are defined according to whether the integrated systems have available interfaces. At last, user-interface integration refers to develop unified user-interfaces or reconstruct existing same or similar user-interfaces.

Keys — Compositionality for Simplicity and Flexibility Constructing complex tools requires being able to understand bigger things from knowing about smaller parts. In other words, we can use fine-grained small services to construct bigger services and even applications under the guidance of established standard. Because the small services would be easier to be understood and used. Consequently, it would make constructing complex meteorological applications simple and flexible.

Logic Workflow Scheduling Webpage Assembling From Fine-Grained to Coarse-Grained Webpage From the Front to the End Logic Workflow Scheduling N1 N2 N3 N4 N5 N6 Start End Workflow Service Assembling Services Data Integration Data Keys — Domain Knowledge Supervised Service Management and Scheduling Engine The fourth is to develop domain knowledge supervised service management and scheduling engine.

Logic Workflow Scheduling Webpage Assembling From Fine-Grained to Coarse-Grained Webpage From the Front to the End Logic Workflow Scheduling N1 N2 N3 N4 N5 N6 Start End Workflow Service Assembling Services Data Integration Data Keys — Domain Knowledge Supervised Service Management and Scheduling Engine The engine should have abilities to disassemble complex services to fine-grained forms and assemble these fine-grained services to coarse-grained services in order to satisfy various domain-specific requirements.

Logic Workflow Scheduling Webpage Assembling From Fine-Grained to Coarse-Grained Webpage From the Front to the End Logic Workflow Scheduling N1 N2 N3 N4 N5 N6 Start End Workflow Service Assembling Services Data Integration Data Keys — Domain Knowledge Supervised Service Management and Scheduling Engine Moreover, the engine should also have abilities to schedule domain-specific logic workflows.

Keys — Domain-Specific Big Data Exploration Environment Domain-Specific Extensibility Domain-Specific Environment Description Keys — Domain-Specific Big Data Exploration Environment The last but not the least, we should construct a domain-specific big data exploration environment. It should have ability to be extended according to meteorological domain-specific requirements, shows environment descriptions that meteorological users can be easily understand, and gives meteorological domain-specific exploration results.

Environment 1：Raw Data Storing To investigate our ideas, we have developed a SOA based environment. It can do meteorological raw data storing.

Environment 2：Visualized Algorithm Development IDE 3：Automatic Code Generation and Optimization Environment develop algorithms in visualized form in the IDE, do automatic code generation and optimization.

Environment 5：Free-Style Domain 4：Graphical Visualization Knowledge Description 4：Graphical Visualization of the Results Environment show data exploration results in graphical visualization form based on meteorological domain-specific requirements, and give a free-style domain knowledge description.

Environment 6：Knowledge Accumulation 7：Rule-based Knowledge Representation Environment save the meteorological domain knowledge to be further used as a rule-based knowledge.

Environment 9：Domain-Specific Standards 8：Domain-Specific Components Moreover, we have developed a lot of meteorological domain-specific service components in the platform, and several human-read meteorological standards have also been implemented in the platform.

Environment 10：Visualized Development Platform Data Acquisition Departure Rate Calculation Domain-Specific Representation Type Setting 10：Visualized Development Platform Environment Domain-Specific Analysis Results Drawing At last, domain-specific application development can be implemented in a visualized way. This is the visualized domain-specific service assembling and logic workflow scheduling canvas. Its underlying implementation conforms to the domain-specific extensions of the general cloud computing models, and it would support meteorological domain-specific application development.

Validation – Landscape Forecasting To validate the effectiveness of the environment, we use our SOA based environment in the project of developing a landscape forecasting system for E-Mei-Shan scenic area. In that system, the meteorological data access, management and integration,

Validation – Landscape Forecasting service assembling,

Validation – Landscape Forecasting logic workflow scheduling,

Validation – Landscape Forecasting and web page assembling and developing are all tested. The development efficiency is improved about seventy percent by using the SOA based meteorological big data exploration environment.

Thanks! Q&A Yaqiang Wang E-Mail: yaqwang@cuit.edu.cn Chengdu University of Information Technology Joint work with members of Joint Research Center for Meteorological Software Engineering, leaded by Prof. Hongping Shu That is all, thanks! Any question?