Huawei Big Data Storage N9000 Jan Stibor Storage & Server Product Manager

1 2 3 Definition of Big Data HUAWEI Big Data Storage Solution Application Scenarios of HUAWEI Solution

Era of Big Data Is Coming 250 million pictures per day Videos of 28,800 hours per day 2.9 million emails per second globally 24 PB data processed per day 200 PB per safe city 6.3 million orders per day More than 1 PB per 3D movie 50 million messages per day 250 million pictures are uploaded to Facebook per day on average. There are 200 million video playbacks on Youku per day. The capacity of a 3D movie is more than 1 PB. The storage capacity of a safe city is 200 PB. Starting from 2012, data amount doubles every two years. By 2020, data amount is 44 times that of 2009. In the first quarter of 2012 alone, the website and mobile client of www.dianping.com gained over 0.8 billion views monthly. During the period when Euro CUP is held, the daily average number of users that visit v.qq.com/ is 1.76 million. If you learn about the big data market composed of servers, storage devices, networks, software, and services defined by IDC, you will discover a market that has a value of 3.2 billion dollars in 2010 and is expected to grow with the compound annual growth rate of 39.4% for five years, reaching 16.9 billion dollars in 2015. IDC said that such a growth rate is seven times that of the entire IT market within the same five years. By 2015, data amount will reach 8 ZB with the valued of $16.9 billion.

What Is Big Data? Mass data Volume Velocity Exponential growth of unstructured data Quick data flow and dynamic data system Real-time analysis, not post analysis Veracity Variety Various data types Unstructured data such as files, emails, and videos accounting for 90% of the data amount generated in the next 10 years  Mass data (volume), quick data flow and dynamic data system (velocity), various data types (variety), and great data value (value) Veracity of Great data Prediction and analysis of the future and the behavior mode IDC: four characteristics of big data

Driving Force of Big Data 1 Storage — open architecture, gradual scale out, and integration of near-line systems and offline systems Big Data Collection 2 Analysis — compound data, unified storage of multiple data sources, unified query, and sharing of storage resources Storage 3 Archiving — lifecycle management of mass data at a low cost and high energy efficiency Big data analysis is a technology-driven strategy that aims to obtain comprehensive, accurate, and in-depth insights into customers, partners, and business, gaining competitive strength. By processing stable real-time data flows, an enterprise is able to promptly make time-sensitive decisions, monitor the latest trend, and adjust its strategy to seize new business opportunities. Big data is a also a big challenge to traditional data storage in terms of data collection > storage > management > cleanup > analysis. Big data requires the following: Big storage: distributed storage and flexible management of mass data High performance: quick access and search Efficient data analysis: real-time analysis, predictive and prescriptive analysis Big analýza dat je technologie-řízený strategie, která si klade za cíl získat komplexní, přesné a do hloubky pohled na zákazníky, partnery a podniky získávají konkurenční sílu. Zpracováním stabilní real-time datové toky, je podnik schopen okamžitě provést časově citlivé rozhodnutí, sleduje nejnovější trendy a přizpůsobit svou strategii využívat nové obchodní příležitosti. Big data jsou také velkou výzvou pro tradiční ukládání dat z hlediska sběru dat> Úložiště> Správa> čištění> analýza. Big dat vyžaduje následující:   Big skladování: distribuované ukládání a flexibilní řízení hromadných dat   Vysoký výkon: rychlý přístup a vyhledávání   Efektivní analýza dat: analýza v reálném čase, prediktivní a normativní analýza Management 4 Maintenance — easy to use Analysis

Contents 1 2 3 Definition of Big Data HUAWEI Big Data Storage Solution Application Scenarios of HUAWEI Solution

HUAWEI Big Data Storage Solution CSS/CSE HUAWEI Big Data Storage Solution Analysis Storage Archiving Large-capacity solution High-performance solution Big Data Mass data query and analysis solution High-performance solution Large-capacity solution Mass data query solution

HUAWEI OceanStor N9000—X in 1 Quick data import Support for JDBC and ODBC SQL 92/2003 Data compression Linear expansion from 3 to 288 nodes More than 3,000,000 OPS Capacity up to 100 PB NFS/CIFS interfaces HDFS interface Global online deduplication Support for LTFS Scale-out NAS Scale-out DB Scale-out Backup

HUAWEI OceanStor N9000 Architecture Application layer Media asset Internet HPC … Behavior prediction, real-time analysis, … Apple …… Unstructured data, object data, structured data Big data sharing BI Big data analysis NAS interface Object interface Archiving interface Database interface Map-reduce HDFS Unified management and namespace Lifecycle policy Index of quick file search Remote replication and snapshot Global deduplication Periodic data inspection HPC: High-Performance Computing Storage Analysis Archiving Storage layer 10GE/Infiniband Node Node Node Node Node Node

N9000 Hardware 1 U high High-performance node： 25 disk slots 2-channel hexa-core CPU 48 GB memory at least SSDs for storing metadata High-performance node： The scene of frequent read and write small files Front view Rear view Front view 4 U high 36 disk slots 2-channel hexa-core CPU 48 GB memory at least High-bandwidth node： The scene of large file sequential read and write Rear view 4 U high 36 disk slots 2-channel quad-core CPU 16 GB memory at least Front view Large-capacity node： Near-line storage Rear view Acceleration node 1 U high

N9000 Software — Wise Series WushanFS Self-developed distributed file system Mandatory, one-off purchase Wise Data Protection (WiseDP) 1 WiseLink Cluster load balancing software Support for multiple load balancing policies Optional, allowing for separate purchase 2 WushanFS WiseQuota Quota management software that sets space usage limits on directories or users. Optional, allowing for separate purchase 3 WushshanFS: a suite of enterprise software program used to provide a single volume, single file system, and a single namespace for forming a base for an enterprise DC. WushanFS: Wushan File System WiseTier DST software Support for policy-based automatic migration Optional, allowing for separate purchase 4

Extensive Flexibility: WiseDP Number of Nodes ODC RDC RDN Max. No. of Allowed Faulty Disks Max. No. of Allowed Faulty Nodes Utilization 3 4 2 1 67% 6 75% 5 8 80% 10 14 12 86% 71% 20 Switch +1 ECC +2 ECC Original data counts (ODC) Redundant data counts (RDC) Redundant data nodes (RDN) +3 ECC +4 ECC N+1 to N+4 data protection File-level data protection 1 hr/TB data recovery

Outstanding Performance: WiseLink Automatic load balancing among existent and newly added nodes ... Single node expansion within 60s Without human intervention and manual modifications

Outstanding Performance: WiseTier Enables DST to work among diverse types of nodes. Adopts tiering policies based on I/O access popularity. Adopts hotspot anti-shake policy to prevent unnecessary data migration. Seamlessly adapts to service changes without interrupting ongoing applications. Consolidates multiple storage systems into a single storage system, eliminating the possibilities of frequent data migration and simplifying management. High-OPS nodes Large-capacity nodes Hot data Cold data

Simplified Management: Refined Space Management WiseQuota: sets quotas for subdirectories to facilitate space management. 100 TB 10 TB 20 TB 40 TB 30 TB Fast automatic deployment Comprehensive performance monitoring User User User group User group Unified topology display 20 TB 10 TB Centralized management of a single file system of 100 PB User User

HUAWEI 9000 — Adapting to Changes and Converging for Easy Management Extensive Flexibility Outstanding Performance Optimal Integration Streamlined Management Scaling from 3 to 288 nodes Support up to 100 PB of capacity Linear scaling of performance and capacity World's leading file system access performance of more than 3 million OPS Distributed scale-out architecture Integration of scale-out NAS, scale-out DB, and scale-out backup Unified management of all nodes on a single interface LCM Adapting to changes and converging for easy management is the core design concept of the N9000. The N9000 provides a powerful scalability from 3 to 288 nodes capable of delivering up 10 PB of storage capacity. The file system access performance is expected to exceed 3 million OPS. The linear scaling of capacity and performance contributes to the unprecedented ultra-large capacity and ultra-high performance. The N9000 innovatively incorporates scale-out NAS, DB, and backup for bearing multiple types of applications and centrally manages all nodes on a single interface for providing complete LCM.

Contents 1 2 3 Definition of Big Data HUAWEI Big Data Storage Solution Application Scenarios of HUAWEI Solution

HUAWEI OceanStor N9000 Scenario— Media Asset Management Focusing on five aspects Animation rendering … Non-linear editing system HD editing SD editing Cataloging TV station non-linear editing system solution Stable bit steams and low single-stream bandwidth, with a strong need for data sharing Animation rendering solution Frame image file storage, with a strong need for high performance in sequential read of small-sized I/Os VOD solution Concurrent reads by multiple users, with a strong need for high bandwidth Media asset management solution Large capacity, high bandwidth, and data sharing Movie post-production solution Stable bid streams and high-bandwidth throughput, providing data sharing ... HPC N9000 high-performance nodes N9000 high-bandwidth nodes and large-capacity nodes VOD server cluster TV set + set-top box PC Smart phone VOD system LAN The solution provides over 2.2 GB/s access throughput to meet the need for design and production of 10 channels. The storage system supports high-speed access to hotspot files. The storage system provides the capability to manage the space and rights of production projects, allowing easy project-level and directory- level space allocation and management of project team members' data access rights. 3.6 GB/s production and rendering performance is provided. Three-level performance acceleration improves hotspot data access performance, allowing 120 rendering computing nodes to access the storage system concurrently. Highly reliable storage is provided, with the support for 7x14 service continuity. The scale-out architecture supports online smooth expansion. Comprehensive capability to manage user rights and space quotas. Snapshot, mirror, and backup provide multi-level continuous protection of critical data. Editing Audio Effect … Query/Cataloging Post-production system Media asset management system