Scale-out Technology, Simplified Meteorological data Lifecycle Management Laijunchen Huawei Technologies Co., Ltd cbs-16@wmo.int

Content New Challenge Practices Future works cbs-16@wmo.int

The whole workflow of meteorological data process Synoptic Analysis Numerical Prediction Climatic Analysis Radar status… satellite Space Ground Station Internal User MPI Flike STORM Ocean Radar Industry User Radiation… Meteorological data pool public User Data acquisition Data analysis and storage Data Service cbs-16@wmo.int

Subscribers incremental What’s happened ? 100x Data explosion New Tools 50x Demand exploding Subscribers incremental cbs-16@wmo.int

How to improve the efficiency? Simplify data life cycle management Unified Data Platform, Avoid data migration Optimize work flow with new tools cbs-16@wmo.int

Data flow in NSMC 4 Times data migration before access 90% data Service time more than 1h FY1 Produce Service Data Archiving ④ ② ① FY2 ③ FC-SAN NAS NAS Tape library Offline tape FY3 cbs-16@wmo.int

26 nodes Single FileSystem 1.3PB data stored Scale-out Storage, Minimize data migration 26 nodes Single FileSystem 1.3PB data stored Data acquisition after 2015 by FY2 Data acquisition after 2008 by FY3 FY1 Produce Service Data Archiving Scale-out Storage ② FY2 ② ① FC-SAN … Tape library FY3 cbs-16@wmo.int

30+% data service time improved Efficiency improvement 30+% data service time improved Service time reduce to 1/3600 cbs-16@wmo.int

Never get the current data if acquisition interval in 1min Challenge for Automatic stations data process Acquisition Analysis Processing Presentation Acquisition frequency 10min to 1min takes 5~10 min to finish whole process Never get the current data if acquisition interval in 1min cbs-16@wmo.int

Optimized data process workflow with JSMB BLU GIS MPPDB BLU Flume MQ(Kafka) Stream OLAP BLU HBase M4 BLU Redis … … Data Service BigData Platform APP cbs-16@wmo.int

Minutes to Seconds Efficiency improvement Milliseconds Acquisition Analysis Processing Presentation Milliseconds User defined logic 50x performance Improvement at history statistics 1M user with auto scaling 10k+ stations Minutes to Seconds cbs-16@wmo.int

Future: Smart meteorological system Unified Meteorological Data Platform Data Fields ① ④ Meteorological Prediction By Machine Learning ② ③ Prediction Service Intelligent Meteorological Data Service Framework Intelligent Elastic Data Service Data Service Gray Release cbs-16@wmo.int

Thank you Merci cbs-16@wmo.int

Reference 1.The Satellite data Archive system of national satellite meteorological center 2.Fengyun Series Meteorological satellite data archiving and service system 3.Design and implementation of china integrated meteorological information sharing system(CIMISS)

30+% data service time shorten to 1/3600 Efficiency improvement 30+% data service time shorten to 1/3600 Hot data Warm data Cold data AS IS Service time <1s <2m 1h~few day Data stored 10% 100% Pre-operation 43%