1. Scale-out Technology, Simplified Meteorological data Lifecycle Management
Laijunchen
Huawei Technologies Co., Ltd

2. Content
New Challenge
Practices
Future works

3. The whole workflow of meteorological data process
Synoptic
Analysis
Numerical
Prediction
Climatic
Analysis
Radar
status…
satellite
Space
Ground
Station
Internal User
MPI
Flike
STORM
Ocean
Radar
Industry User
Radiation…
Meteorological data pool
public User
Data acquisition
Data analysis and storage
Data Service

4. Subscribers incremental
What’s happened ?
100x
Data explosion
New Tools
50x
Demand exploding
Subscribers incremental

5. How to improve the efficiency?
Simplify data life cycle management
Unified Data Platform, Avoid data migration
Optimize work flow with new tools

6. Data flow in NSMC
4 Times data migration before access 90% data Service time more than 1h
FY1
Produce
Service
Data Archiving ④ ② ①
FY2 ③
FC-SAN
NAS
NAS
Tape library
Offline tape
FY3

7. 26 nodes Single FileSystem 1.3PB data stored
Scale-out Storage, Minimize data migration
26 nodes Single FileSystem
1.3PB data stored
Data acquisition after 2015 by FY2
Data acquisition after 2008 by FY3
FY1
Produce
Service
Data Archiving
Scale-out Storage ②
FY2 ② ①
FC-SAN …
Tape library
FY3

8. 30+% data service time improved
Efficiency improvement
30+% data service time improved
Service time reduce to 1/3600

9. Never get the current data if acquisition interval in 1min
Challenge for Automatic stations data process
Acquisition
Analysis
Processing
Presentation
Acquisition frequency
10min to 1min
takes 5~10 min to finish whole process
Never get the current data if acquisition interval in 1min

10. Optimized data process workflow with JSMB
BLU
GIS
MPPDB
BLU
Flume
MQ(Kafka)
Stream
OLAP
BLU
HBase M4
BLU
Redis … …
Data
Service
BigData Platform
APP

11. Minutes to Seconds Efficiency improvement Milliseconds
Acquisition
Analysis
Processing
Presentation
Milliseconds
User defined logic
50x performance
Improvement at history statistics
1M user
with auto scaling
10k+ stations
Minutes to Seconds

12. Future: Smart meteorological system
Unified Meteorological Data Platform
Data
Fields ① ④
Meteorological Prediction By Machine Learning ② ③
Prediction
Service
Intelligent Meteorological Data Service Framework
Intelligent Elastic Data Service
Data Service Gray Release

13. Thank you
Merci

14. Reference
1.The Satellite data Archive system of national satellite meteorological center 2.Fengyun Series Meteorological satellite data archiving and service system 3.Design and implementation of china integrated meteorological information sharing system(CIMISS)

15. 30+% data service time shorten to 1/3600
Efficiency improvement
30+% data service time shorten to 1/3600
Hot data
Warm data
Cold data
AS IS
Service time
<1s
<2m
1h~few day
Data stored
10%
100%
Pre-operation
43%