Lustre use cases and Virtualization of Lustre

Slides:



Advertisements
Similar presentations
Ddn.com ©2012 DataDirect Networks. All Rights Reserved. GridScaler™ Overview Vic Cornell Application Support Consultant.
Advertisements

Engenio 7900 HPC Storage System. 2 LSI Confidential LSI In HPC LSI (Engenio Storage Group) has a rich, successful history of deploying storage solutions.
February 6, 2014 Ambuj Goyal General Manager, System Storage & Networking IBM Systems and Technology Group Why Infrastructure Matters.
© 2007 DataDirect Networks, Inc. All Rights Reserved. Simple Petabyte-Level Scaling Industry Leading Performance SATAssure Data Integrity Power & Space.
Cisco and NetApp Confidential. Distributed under non-disclosure only. Name Date FlexPod Entry-level Solution FlexPod Value, Sized Right for Smaller Workloads.
© Hitachi Data Systems Corporation All rights reserved. 1 1 Det går pænt stærkt! Tony Franck Senior Solution Manager.
1© Copyright 2013 EMC Corporation. All rights reserved. EMC and Microsoft SharePoint Server Performance Name Title Date.
© 2013 Mellanox Technologies 1 NoSQL DB Benchmarking with high performance Networking solutions WBDB, Xian, July 2013.
© Copyright 2010 Hewlett-Packard Development Company, L.P. 1 HP + DDN = A WINNING PARTNERSHIP Systems architected by HP and DDN Full storage hardware and.
FlashSystem family 2014 © 2014 IBM Corporation IBM® FlashSystem™ V840 Product Overview.
Tag line, tag line NetApp HPC Update 28 th HPC User Forum April 2008 Norfolk, Virginia.
March 9, 2015 San Jose Compute Engineering Workshop.
© 2010 DataDirect Networks. Confidential Information D D N D E L I V E R S DataDirect Networks Update Mike Barry, HPC Sales April IDC HPC User Forum.
Bosch DSA Storage (based on NetApp E2700)
High Performance Storage Solutions April 2010 Larry Jones VP, Product Marketing.
Axis AI Solves Challenges of Complex Data Extraction and Document Classification through Advanced Natural Language Processing and Machine Learning MICROSOFT.
Tackling I/O Issues 1 David Race 16 March 2010.
Built on the Powerful Microsoft Azure Platform, Forensic Advantage Helps Public Safety and National Security Agencies Collect, Analyze, Report, and Distribute.
High Performance Storage System (HPSS) Jason Hick Mass Storage Group HEPiX October 26-30, 2009.
Extreme Scale Infrastructure
E2800 Marco Deveronico All Flash or Hybrid system
EonStor DS 1000.
AuraPortal Cloud Helps Empower Organizations to Organize and Control Their Business Processes via Applications on the Microsoft Azure Cloud Platform MICROSOFT.
Univa Grid Engine Makes Work Management Automatic and Efficient, Accelerates Deployment of Cloud Services with Power of Microsoft Azure MICROSOFT AZURE.
Video Security Design Workshop:
Organizations Are Embracing New Opportunities
Achieving the Ultimate Efficiency for Seismic Analysis
The demonstration of Lustre in EAST data system
New Heights by Guiding Them into the Cloud
What is Cloud Computing - How cloud computing help your Business?
Free Cloud Management Portal for Microsoft Azure Empowers Enterprise Users to Govern Their Cloud Spending and Optimize Cloud Usage and Planning MICROSOFT.
Trial.iO Makes it Easy to Provision Software Trials, Demos and Training Environments in the Azure Cloud in One Click, Without Any IT Involvement MICROSOFT.
Appro Xtreme-X Supercomputers
A10 Networks vThunder Leverages the Powerful Microsoft Azure Cloud Platform to Offer Advanced Layer 4-7 Networking, Security on a Global Scale MICROSOFT.
Couchbase Server is a NoSQL Database with a SQL-Based Query Language
Veeam Backup Repository
Large Scale Test of a storage solution based on an Industry Standard
SmartHOTEL Solutions Powered by Microsoft Azure Provide Hoteliers with Comprehensive, One-Stop Automated Management of All Booking Channels MICROSOFT AZURE.
OpenNebula Offers an Enterprise-Ready, Fully Open Management Solution for Private and Public Clouds – Try It Easily with an Azure Marketplace Sandbox MICROSOFT.
Built on the Powerful Microsoft Azure Platform, Lievestro Delivers Care Information, Capacity Management Solutions to Hospitals, Medical Field MICROSOFT.
Microsoft Azure Platform Powers New Elements Constellation Software Suite to Deliver Invaluable Insights From Your Data for Marketing and Sales MICROSOFT.
Oscar AP by Massive Analytic: A Precognitive Analytics Platform for Effortless Data-Driven Decisions. Now Available in Azure Marketplace MICROSOFT AZURE.
Built on the Powerful Microsoft Azure Platform, iSwarm Helps Businesses Analyze Social Media Conversations, then Connect with Individuals MICROSOFT AZURE.
Scalable SoftNAS Cloud Protects Customers’ Mission-Critical Data in the Cloud with a Highly Available, Flexible Solution for Microsoft Azure MICROSOFT.
The Only Digital Asset Management System on Microsoft Azure, MediaValet Is Uniquely Equipped to Meet Any Company’s Needs MICROSOFT AZURE ISV PROFILE: MEDIAVALET.
Zadara™ Virtual Private Storage Arrays™: High Performance, High Availability NAS & SAN by the Hour, with Private, Dedicated Resources MICROSOFT AZURE.
Data Security for Microsoft Azure
CloudLabs, Powered by Azure, Enables the Quick, Easy, Cost-Effective Management, Distribution of Online Training Labs for Education and Business MICROSOFT.
Unitrends Enterprise Backup Solution Offers Backup and Recovery of Data in the Microsoft Azure Cloud for Better Protection of Virtual and Physical Systems.
CloneManager® Helps Users Harness the Power of Microsoft Azure to Clone and Migrate Systems into the Cloud Cost-Effectively and Securely MICROSOFT AZURE.
Dell Data Protection | Rapid Recovery: Simple, Quick, Configurable, and Affordable Cloud-Based Backup, Retention, and Archiving Powered by Microsoft Azure.
One-Stop Shop Manages All Technical Vendor Data and Documentation and is Globally Deployed Using Microsoft Azure to Support Asset Owners/Operators MICROSOFT.
Appcelerator Arrow: Build APIs in Minutes. Connect to Any Data Source
Media365 Portal by Ctrl365 is Powered by Azure and Enables Easy and Seamless Dissemination of Video for Enhanced B2C and B2B Communication MICROSOFT AZURE.
XtremeData on the Microsoft Azure Cloud Platform:
Quasardb Is a Fast, Reliable, and Highly Scalable Application Database, Built on Microsoft Azure and Designed Not to Buckle Under Demand MICROSOFT AZURE.
Improve Patient Experience with Saama and Microsoft Azure
Technical Capabilities
Harness the competitive advantages of Power BI and obtain business-critical insights with Adastra’s enterprise analytics platform using Microsoft Azure.
BluSync by ParaBlu Offers Secure Enterprise File Collaboration and Synchronization Solution That Uses Azure Blob Storage to Enable Secure Sharing MICROSOFT.
Guarantee Hyper-V, System Center Performance and Autoscale to Microsoft Azure with Application Performance Control System from VMTurbo MICROSOFT AZURE.
DATS International Portfolio.
On the Role of Burst Buffers in Leadership-Class Storage Systems
Pitch Deck.
Microsoft Virtual Academy
Microsoft Virtual Academy
COMPANY PROFILE: REELWAY
How Dell, SAP and SUSE Deliver Value Quickly
OpenStack for the Enterprise
Presentation transcript:

Lustre use cases and Virtualization of Lustre Philip Zhu Director, North Asia

DDN | The Leader In Massively Scalable Storage Solutions & Services Key Statistics Industry Validation Delivers highly scalable and highly efficient storage solutions that enable customers to accelerate time to results, scale simply as data sets continue to grow, and gain competitive advantage through resolving performance and capability scaling challenges Established: 1998 Financials: Over $200M Annually, Very Profitable Headquarters: Chatsworth, California USA Employees: Approximately 400 Worldwide Customers: Over 1,000 Worldwide Footprint: 17 Industries, 4 Continents, 49 Countries Go to Market: Global Partners, VARs, Resellers Key Market Segments: High Performance Computing & Life Science Cloud & Web Content Rich Media Intelligence/Federal Surveillance World’s Largest Privately-Held Storage Co. Fast500 Technology Company Inc. 500|5000 High-Growth Company Best HPC Storage Platforms Best Practice for Digital Media 1000+ World-Leading Customers

DDN HPC Storage Market Leadership Powering 60% of the Top50 fifty largest systems More than all other technology companies combined Storage for over 1/3 of compute power for the Top500 More than 19,500 TeraFlop/s – more than any other technology company Deployed in 118 of the worlds fastest supercomputers More than half of the Top 100 Fastest file systems in the world – over 300GB/s! Wright-Patterson AFB >300GB/s CEA Tera-100 at 300GB/sec Sample Customers

Worldwide Performance Leader The World's Fastest Parallel File System RDMA Capable Single Client throughput of 2.5GB/s+ Ideal for 10,000s of concurrent client access to storage Supports full single file concurrency Supports full single directory concurrency Network Flexible Writes Natively to IP / IB S2A Real-time storage technology provides guaranteed performance Writes are as fast as Reads No degradation in performance due to drive errors, rebuild events or enclosure failures DDN Exascaler Technology Powers the World's Fastest File Systems: 350GB/s at CEA 250GB/s as US DoD 250GB/s at ORNL

HPC | The Worldwide Storage Leader 100s of Leading HPC Customers Worldwide Deploying The Fastest Systems On The Planet Over 50% of The Top100 And many, many more…

Accelerating Accelerators DDN is the leading provider of enterprise-grade storage platforms for the next generation of particle physics research. DDN Supplied Over 30PB of Atlas Storage since 2009

Success Story: LLNL With a combined 1.6 Petaflop/s of peak performance and over 350GB/s of aggregate DDN storage bandwidth, LLNL is the world's largest data-intensive computing facility. Since 2005, DDN has delivered world-class solutions and expertise to help LLNL scale computational capabilities by over 1000x.

Oakridge National Labs – Jaguar Super Computer The HPC storage leader, DDN maintains advantage through hyper-scale expertise At 240GB/s – DDN has delivered the world's most scalable bio-informatics storage solution to ORNL ORNL has over 400GB/s of DDN performance & capacity efficient solutions Today's supercomputers are tomorrow's workgroup clusters, only DDN technology scales up and down to meet and grow with the complex requirements of life sciences Genome Analysis and Systems Modeling Group Currently Available Tools at ORNL GrailEXP Gene finder (Human and Mouse Genomes) Generation Gene finder (Microbial Genomes) Pipeline, Comprehensive Genome Analysis Pipeline Genome Channel, Java applet for the comprehensive sequence-based view of genomes. Grail, a tool for the identification of genes, exons, and various features in DNA sequences. The Computational Biology and Bioinformatics Group at Oak Ridge National Laboratory conduct genetics research and system development in genomic sequencing, computational genome analysis, and computational protein structure analysis. They provide bioinformatics and analytic services and resources to collaborators, predict prospective gene and protein models for analysis, provide user services for the general community, including computer-annotated genomes in Genome Channel. Collaborators include: - The Joint Genome Institute, - ORNL's Computer Science and Mathematics Division, - The Tennessee Mouse Genome Consortium, - The Joint Institute for Biological Sciences, and - ORNL's Genome Science and Technology Graduate Program. Gating Mechanism of Membrane Proteins PI: Benoit Roux, Argonne National Laboratory/University of Chicago Physical Basis of Recalcitrance to Hydrolysis of Lignocellulosic Biomass Principal Investigator: Jeremy Smith, ORNL Enabling Petascale Research

Success Story: ORNL When ORNL needed to build the world's fastest POSIX file system to support the world's fastest supercomputer, DDN delivered. S2A9900 delivered the highest performance in ORNL testing while also enabling striped file I/O (with DDN QOS) and leading density.

Success Story: ORNL Jaguar (XT5) Jaguar (XT4) ESnet, USN, TeraGrid, Internet2, NLR Jaguar (XT4) SeaStar Torus SeaStar Torus 10 – 40 Gbit/s HPSS Archive (10 PB) Smokey Lens GridFTP Servers Lustre-WAN Gateways RTR RTR RTR RTR RTR 192 Routers 48 Routers Scalable I/O Network (SION) - DDR InfiniBand – 889 GB/s OSS OSS OSS OSS OSS OSS OSS 192 OSSs 1344 OSTs

Spider File System: Facts 253 GB/s of Aggregate Write Bandwidth 48 DDN 9900 Couplets 13,440 1TB SATA Drives Over 10 PB of RAID6 Capacity 192 Storage Servers Over 1000 InfiniBand Cables ~0.5 MW of Power ~20,000 LBs of Disks Fits in 32 Cabinets using 572 ft2

Architecture and Implementation of Lustre at the National Climate Computing Research Center

TSUBAME2.0 storage

Use Case: TOTAL DDN was Chosen Because : ExaScaler provides high performance concurrent client access which allowed all clients to access and read the same set of source files. Up to 250GB/s file Storage throughput Writes natively to IB RDMA capable ExaScaler scalability allows for TOTAL’s future capacity planning  Highly scalable, highly-efficient storage system ExaScaler leverages commodity technology DDN’s parallel architecture enables leading lustre performance 14

Markets ExaScaler plays in Supercomputing Labs Powering the World's Fastest HPC System, File System over 300 GB/sec Provides better ROI by ensuring clusters spend more time computing and less time doing I/O. Academic Computing/Research Universities DDN systems are capable of handling a mixed workload, allowing many researchers to work simultaneously and reduce research cycle times. DDN HPC systems provide superior $/performance to ensure optimum HPC TCO. Oil & Gas DDN shortens "time to oil" through accelerating seismic processing algorithms DDN Systems provide per-client performance and scale to allow customers to support complex 4D modeling and interpretation of seismic surveys. Life Sciences and Genomics Enable sequencing centers and bioinformatics departments to accelerate their development pipeline, reduce time to market for new drugs Reduce the Gene Sequencing pipeline by up to 30% by using native clients Government Intelligence World's fastest storage systems for image/motion capture, in-theater reconnaissance DDN’s real-time, scalable technology ensures rapid ingest and immediate data availability to ensure real-time insight and the shortest possible kill chain.

Storage Fusion Architecture™ SFA1XK-X™

Supercharged Building Block ExaScaler is built with Open Source Lustretm Technology 100+ 75% Lustre powers more than 100+ of the Top 500 Supercomputing sites of these 100+ sites run DDN storage in the backend ExaScaler draws from the experience of the world’s Leading Lustre Deployments

ExaScaler-At a Glance 10s to 1000s of Linux HPC Clients 1Gb / 10Gb / InfiniBandTM 10s to 1000s of Linux HPC Clients Intelligently Scale Applications Intelligently Manage Infrastructure Object-Based File System Enables Granular Read/Write Access And Massively Concurrent Access Non Disruptive Scaling Multi-Tiered File System Out-Of-Band Metadata Server Cluster Can Handle 10,000s of Operations Per Second Integrated Backup and HSM Tools Parallel Configuration Enables Simple Scaling Multi-Petabyte, Scalable Parallel Storage System No-Compromise Scale Out Performance & Data Protection 100s of GB/s of Performance, Linear Performance Scaling & Leading Data Center Efficiency

DDN | Scale With Flexibility Scale-Up For Capacity Intelligent, Scale-Out File Systems Aggregate Clustered Storage For Speed & Scale Flexible Scalability Is a unique capability of DDN's whereby we can deliver maximum system-level efficiency to an environment by never shoehorning a customer into a rigid configuration. With traditional scale-up NAS (a la NetApp), customers are forced to deploy silos of non-scalable systems where the lack of a global namespace forces a data management issue within the data center. These systems are often suitable for capacity optimized environments, but cannot scale capacity even beyond a single system. On the flip side, scale-out technologies are often times over-built with excess CPU, memory, systems and licenses for capacity-intensive applications. Systems such as Isilon's Scale-Out cluster technology can deliver decent performance (1/10th of DDN's delivered performance in high end HPC environments), but the cost of this scaling is punitive for customers looking for a capacity optimized solution. At DDN...we right-size systems based upon customer requirements to scale-up when capacity is needed, and scale out when performance or a combination of performance and capacity is needed. The result is a dramatically lower TCO than our competitors because we never have to sell more than what a customer needs. And because the customer has invested in the scalability leader – they can be sure that however their requirements evolve – we have the tools to grow with these changes simply, efficiently and intelligently. Each System Is A Scalable Building Block, Supporting Up To 1,680 Disks To Enable Cost-Efficient Capacity & Simple Configuration Scale-Out For Performance

Scalable Building Block Architecture Allows Seamless and Simple Expansion on Demand Offers Diagonal Scalability - Scale Up as well as Scale Out Individually scale Performance and/or Capacity based on needs No hidden licensing or configuration barriers to complicate growth Build a solution based on your budget - Predictable Growth and TCO Scale Out Clustered High Performance File Services Scale Up DDN Storage Fusion Architecture Storage Appliance DDN High-Density Mixed-Media Disk Enclosure

The SFA12K Family SFA12K-20 (Block Appliance) Highly Parallelized SFA Storage Processing Engine Active/Active Storage Design ~40GB/s Read & Write Speed Up to 3.3PB of Disk 1.2+ Million Burst IOPS 700K+ Random Spinning Disk IOPS 850K+ Sustained Random SSD IOPS 32GB+ Mirrored Cache (Protected) RAID 1/5/6 Intelligent Block Striping SATAssure Data Protection GUI, SNMP, CLI, API 8 x FDR IB Host-Ports 8RU Height 16 x 16Gb/s Fibre Channel Host Ports 8 x FDR InfiniBand Host Ports SFA Interface Virtualization SFA Interface Virtualization 16-32GB High-Speed Cache 120Gb/s Cache Link 16-32GB High-Speed Cache Internal SAS Switching Internal SAS Switching 480Gb/s Internal SAS Storage Management Network SFA RAID 5,6 RAID 5,6 SFA RAID 1 1 2 3 4 5 6 7 8 P Q RAID 6 1m

SFA10000 Embedded ExaScaler

HPC Storage on the SFA10000E Appliance

SFA10K-E | Infrastructure Efficiency Parallel File System Clients SFA10K-E with embedded File Systems can result in a 10 to 1 or greater reduction in managed systems. Storage Fusion Architecture not only reduces complexity, but also streamlines IO by reducing latency and protocol conversion 1 Scalable Building Block Which Incorporates File Services And Eliminates Gateways & Networking for Scale Out

3.5” & 2.5” SSD, SAS & SATA (inter-mixable) SFA12K™ | Models ™ SFA12K-20 SFA12K-20E SFA12K-40 Maximum Drives 1,6801 1,6801 1,6801 FDR IB 16Gb FC2 FDR IB 10/40GbE FDR IB 16Gb FC2 System Interface Drive Types 3.5” & 2.5” SSD, SAS & SATA (inter-mixable) System Capacity 6.72PB (w/ 4TB HDDs)1 20GB/s (raw I/O) 20GB/s (file I/O) 40GB/s (raw I/O) Bandwidth Cache IOPS 850K 850K 1.7M Flash IOPS 700K 700K 1.4M In-Storage Processing™ Yes. ExaScaler, GridScaler Customer Provided N/A N/A 1 840 Drives Until 2H12 2 16Gb FC available 2H12

DDN | First in In-Storage Computing Block Storage Appliance File Storage Appliance Open Computing Appliance SFA12K™-20/40 Block Storage Target SFA12K-20E Parallel File Storage EXAScaler™ GRIDScaler™ SFA12K-20E Customer Applications Pre-Processing Post-Processing Flexible Deployment Options: 3 System Modalities

Storage + Processing | Converged Storage Fusion Processing™ SFA12K-20E Systems Feature Up to 16 Virtual Machines Support for Linux and Microsoft Windows applications Highly-optimized virtualization delivers up to 96% efficiency. Each VM gets dedicated networking for bandwidth & lowest latency VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM 120Gb/s Cache Link

SFA12K™-20E | Parallel File Storage Appliances SFA12K-20E available with DDN | EXAScaler™ and DDN | GRIDScaler™ parallel file storage solutions Integrate multiple appliances to scale to over 1000GB/s and 10’s of petabytes EXAScaler SFA12K-20E 20GB/s Up To 5.3PB* Usable capacity GRIDScaler SFA12K-20E 20GB/s Up To 5.3PB* Usable capacity * - Initial release limited to 840 Drives

Thank You pzhu@ddn.com 13321177500 (中国) +81-80-3518-9101 (全球)