0. Scale-out Storage, Simplified Meteorological data Lifecycle Management
Laijunchen
Market manager Cloud Storage Domain
Huawei IT product line

1. MERANTI Caused More Than 1.5 B$ Damage
96 People
be Killed /wounded /disappear
2.56 B$
Property has been damaged

2. Could "Tornado Hunter" Become A Reality?
<Globe lightning>
Could "Tornado Hunter" Become A Reality?
At least 5 days in advance to predict whether the typhoon will cause serious disaster
Sufficient condition:
Global Data Collection
Base on Decades data
刘慈欣 2015

3. How to improve the efficiency of Hundreds PB’s data processing?
Could The Existing Meteorological Data Processing Architecture Achieve That Goal?
Hot Data (Online)
Data in Silos
too difficult to use the data
Cold data (Offline)
Archive
FC-SAN
Tap
only 2-4 Weeks data
Retrieve
More than 20 years data
Expensive
Hard to scale
Poor performance
Hard to manage
How to improve the efficiency of Hundreds PB’s data processing?
Is that be affordable?

4. The perfect Data Management Architecture
Unified Data Pool
A unified namespace can directly provide data storage and data sharing service;
Easy to expand the capacity, and Store huge amounts of data with lower TCO;
All data can been accessed directly at whole lifecycle, so the business systems may access to any range of spatial and temporal data as input as they need.
Hot Data (SSD)
Automatic data tiering
Warm Data
(HDD: Last 3 Years data )
Be Accessed Directly
Cold Data
(High density, energy saving?
Other data）
New Meteorological data lifecycle Model

5. Retrieve before access
Scale-out Storage, Simplified Meteorological Data Lifecycle Management (Phase 1)
Hot Data (Online)
Warm Data(Online)
Cold data(Offline)
Archive
Scale-out Storage
FC-SAN …
Tap
only 2-4 Weeks data
Last 3 Years data
More than 20 years data
Retrieve before access
Middle range TCO
Easy to use, Scale and manage
Poor performance
Hard to manage
Expensive
Hard to scale

6. Hundreds PB Unified Namespace
Data on demand: Hundreds PB pool, Data Always Online
Media
CCTV
HPC
Web Disk
Archive
Behavior
Analysis
Precision
marketing
Business
promotion
Apps
File
Object
Big data
NFS
CIFS
FTP
Swift S3
HDFS
File Service
Object Service
Big data Service
Software
Hundreds PB Unified Namespace
Hardware
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
100 PB unified namespace can provide data storage and data sharing service

7. Flexible Network and Diversify Hardware
Servers
Front end network
Nodes of scale out storage
Throughput
Cost per TB
C72
C36
P36
P12
P25
Front end network
P25 2U 25Bay
C72 4U 72Bay
10GE/Infiniband supported both front-end and back-end network
Diversify Hardware balance performance, capacity, and cost

8. Unified Namespace, Let Data Management Easier
Capacity quota
File quantity quota
100TB
Soft quota
Hard quota
Numerical forecast
Pressure analysis
User quota
Directory quota
Cloud chart
User group quota
40 TB
5 PB
200 TB
Varied quota policies makes space Management easier
NIS, AD, LDAP domain control, translate to Business centric data management.
Thin provisioning.
Temperature
50 TB
6TB
10TB
8TB
5TB
User a
User b
User c
User d
Business centric data management, all data can be accessed form Standard interface

9. Distributed RAID (EC) Provide High Availability and Excellent Performance
400GB/s +1 +3 +1 +2 +4 +2
Capacity:100 PB +3
4 node failure tolerance, 1TB/Hr Rebuild speed
3 to 288 nodes, up to 400GB/s throughput

10. Data Tiering, Balance The Performance And Cost
SSD
Creation time
Access time
File size
I/O popularity
File name
I/O count
SAS
P Serial Hardware
Modification time
User-defined
SATA
The value of data determines its storage location, achieving optimal resource configuration.
Free and transparent data flowing
Ample policies precisely divide tiers.
C Serial Hardware

11. Success Story At NMSC(China)
FY3C
99 Nodes
FY3D
5.5 PB Capacity
20% Improvement

12. Scale-out Storage, Simplified Meteorological Data Lifecycle Management (Phase 2)
Using High Performance Node to store Hot data to replace the High-end FC-SAN
Huawei will release All Flash Node for Scale-out Storage with 5GB/s Bandwidth and 30K OPS soon
Automatic data tiering between hot-data-layer and warm-data-layer
Automatic
tiering
Hot Data (SSD)
Warm Data
(HDD: Last 3 Years data )
Scale-out
Storage
Archive
Retrieve
Cold data
(More than 20 years data)
Tape

13. Scale-out Storage, Simplified Meteorological Data Lifecycle Management (Phase 3)
Automatic tiering
Hot Data (SSD)
Warm Data
(HDD: Last 3 Years data )
Scale-out
Storage
All data managed by Scale-out Storage
Data could archive to Public cloud and transparent to Applications
Cold Data
(High density）
Automatic Archive/Retrieve
Public Cloud

14. Huawei Scale-out Storage Architecture Evolution, Makes Data Manage Easier
2016
Block/File/Object as a Service
(Control plane)
2015 + +
Distributed
Block
Converged Distributed Block/File/Object Storage
Distributed
File
Distributed
Object
(Data plane)
Commodity Server
2012
2013
(Hardware)
2014
Decouple Control and Data Plane
Decouple Software and Hardware
Commodity Server

15. Huawei Scale-out Storage Technology Evolution
World first NAS with
25Gb Ethernet
Fully Converged
Cloud Storage
All Flash Node for Scale-out Storage
100Gb Ethernet,32TB SSD Release
2015
2016
2017