Limitless Storage, Boundless Opportunities Technology Overview – January 2009.

Slides:



Advertisements
Similar presentations
Archive Task Team (ATT) Disk Storage Stuart Doescher, USGS (Ken Gacke) WGISS-18 September 2004 Beijing, China.
Advertisements

Distributed Data Processing
Communication and Networking Services Networking Services.
NAS vs. SAN 10/2010 Palestinian Land Authority IT Department By Nahreen Ameen 1.
MUNIS Platform Migration Project WELCOME. Agenda Introductions Tyler Cloud Overview Munis New Features Questions.
Chapter 9: Moving to Design
Nikolay Tomitov Technical Trainer SoftAcad.bg.  What are Amazon Web services (AWS) ?  What’s cool when developing with AWS ?  Architecture of AWS 
What is it? CLOUD COMPUTING.  Connects to the cloud via the Internet  Does computing tasks, or  Runs applications, or  Stores Data THE AVERAGE CLOUD.
Storage Area Network (SAN)
Data Centers and IP PBXs LAN Structures Private Clouds IP PBX Architecture IP PBX Hosting.
Hardware and Software Basics. Computer Hardware  Central Processing Unit - also called “The Chip”, a CPU, a processor, or a microprocessor  Memory (RAM)
Servers Redundant Array of Inexpensive Disks (RAID) –A group of hard disks is called a disk array FIGURE Server with redundant NICs.
Session 3 Windows Platform Dina Alkhoudari. Learning Objectives Understanding Server Storage Technologies Direct Attached Storage DAS Network-Attached.
Virtual Network Servers. What is a Server? 1. A software application that provides a specific one or more services to other computers  Example: Apache.
1 NETE4631 Communicating with the Cloud and Using Media and Streaming Lecture Notes #14.
Copyright © 2012 Cleversafe, Inc. All rights reserved. 1 Combining the Power of Hadoop with Object-Based Dispersed Storage.
Computerized Networking of HIV Providers Networking Fundamentals Presented by: Tom Lang – LCG Technologies Corp. May 8, 2003.
November 2009 Network Disaster Recovery October 2014.
D-Link International Private Limited Training and Staff Development Department Module : Network Attached Storage Module : Network Attached Storage.
Introduction to Computers
Opensource for Cloud Deployments – Risk – Reward – Reality
Chapter 9 Elements of Systems Design
By Mihir Joshi Nikhil Dixit Limaye Pallavi Bhide Payal Godse.
Overview of SQL Server Alka Arora.
Technology Overview. Agenda What’s New and Better in Windows Server 2003? Why Upgrade to Windows Server 2003 ?  From Windows NT 4.0  From Windows 2000.
Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill Technology Education Copyright © 2006 by The McGraw-Hill Companies,
Hosted by Case Study - Storage Consolidation Steve Curry Yahoo Inc.
IT Infrastructure Chap 1: Definition
Lector: Aliyev H.U. Lecture №15: Telecommun ication network software design multimedia services. TASHKENT UNIVERSITY OF INFORMATION TECHNOLOGIES THE DEPARTMENT.
Virtualization in the NCAR Mass Storage System Gene Harano National Center for Atmospheric Research Scientific Computing Division High Performance Systems.
Chapter 9 Section 2 : Storage Networking Technologies and Virtualization.
What is the cloud ? IT as a service Cloud allows access to services without user technical knowledge or control of supporting infrastructure Best described.
Chapter 1 Computer Systems. Why study Computer Architecture? Examples Web Browsing - how does the browser access pages from a server? How can we create.
Kiew-Hong Chua a.k.a Francis Computer Network Presentation 12/5/00.
NETWORK HARDWARE AND SOFTWARE MR ROSS UNIT 3 IT APPLICATIONS.
1 MSCS 237 Overview of web technologies (A specific type of distributed systems)
9 Systems Analysis and Design in a Changing World, Fourth Edition.
1 NETE4631 Working with Cloud-based Storage Lecture Notes #11.
9 Systems Analysis and Design in a Changing World, Fourth Edition.
11 CLUSTERING AND AVAILABILITY Chapter 11. Chapter 11: CLUSTERING AND AVAILABILITY2 OVERVIEW  Describe the clustering capabilities of Microsoft Windows.
Virtual Machines Created within the Virtualization layer, such as a hypervisor Shares the physical computer's CPU, hard disk, memory, and network interfaces.
I NTRODUCTION TO N ETWORK A DMINISTRATION. W HAT IS A N ETWORK ? A network is a group of computers connected to each other to share information. Networks.
Architecture & Cybersecurity – Module 3 ELO-100Identify the features of virtualization. (Figure 3) ELO-060Identify the different components of a cloud.
Copyright © 2012 Cleversafe, Inc. All rights reserved. 1 Combining the Power of Hadoop with Object-Based Dispersed Storage.
COMPUTER SYSTEM A computer system is define as combination of components designed to process data and store files. A computer system consists of four.
Hands-On Microsoft Windows Server 2008 Chapter 7 Configuring and Managing Data Storage.
The Internet Technological Background. Topic Objectives At the end of this topic, you should be able to do the following: Able to define the Internet.
Office 365 is cloud- based productivity, hosted by Microsoft. Business-class Gain large, 50GB mailboxes that can send messages up to 25MB in size,
July 30, 2009opsarea meeting, IETF Stockholm1 Operational Deployment and Management of Storage over the Internet David L. Black, EMC IETF opsarea meeting.
An Introduction to Local Area Networks An Overview of Peer-to-Peer and Server-Based Models.
9 Systems Analysis and Design in a Changing World, Fifth Edition.
Amazon Web Services. Amazon Web Services (AWS) - robust, scalable and affordable infrastructure for cloud computing. This session is about:
What is raid? RAID is the term used to describe a storage systems' resilience to disk failure through the use of multiple disks and by the use of data.
Fault – Tolerant Distributed Multimedia Streaming Web Application By Nirvan Sagar – Srishti Ganjoo – Syed Shahbaaz Safir
WHAT IS CLOUD COMPUTING? Pierce County Library System.
A Solution for Maintaining File Integrity within an Online Data Archive Dan Scholes PDS Geosciences Node Washington University 1.
Course: Cluster, grid and cloud computing systems Course author: Prof
Video Security Design Workshop:
WEBINAR Introduction to ViconNet 3.0 October 26, 2005
Amazon Storage- S3 and Glacier
Storage Networking.
SAN and NAS.
Introduction to Networks
Introduction to Networks
Storage Networking.
Scalable SoftNAS Cloud Protects Customers’ Mission-Critical Data in the Cloud with a Highly Available, Flexible Solution for Microsoft Azure MICROSOFT.
An Introduction to Computer Networking
Distributed Systems Bina Ramamurthy 11/30/2018 B.Ramamurthy.
Distributed Systems Bina Ramamurthy 12/2/2018 B.Ramamurthy.
Cost Effective Network Storage Solutions
Presentation transcript:

Limitless Storage, Boundless Opportunities Technology Overview – January 2009

Cleversafe Mission Enable the world to confidently store and distribute limitless data Commercial Products Providing Products and Service to companies who are building and operating Dispersed Storage Networks Dispersed Storage - Open Source Project Creating a genuine open standard and a broad technical community 2

3 Data Storage Growth Traditional DataAdditional, New Data Digital Content - 85% of all data by Growing 10x every 4 years Images – 500KB per picture Audio – 5,000 KB per song Video – 5,000,000 KB per movie Documents Character & numerical databases + Source: IDC

4 Current High Availability Scenario 300% Disk Storage Overhead + Tape Backup -Total bytes stored = 4x usable capacity 200% Bandwidth Overhead -Each node supports full operational requirement -Total bandwidth required = 3x operational requirement Higher Cost - More Power - More Management - More Space - More Equipment More Security Risks ParityA3 RAID3 Controller Location 1 A2 A1 The quick brown fox jumps over the lazy brown dog Internet Connection Internet Connection Internet Connection ParityA3 RAID3 Controller Location 2 A2 A1 The quick brown fox jumps over the lazy brown dog ParityA3 RAID3 Controller Location 3 A2 A1 The quick brown fox jumps over the lazy brown dog

5 Digital Data Storage - An Antiquated Approach Currently Data Storage = Data Copies Not Secure – 200 major announced security breaches since 2004 Not Private – Data copies are… data copies Not Long Term – Tied to hardware which doesn’t last over 5 years More Reliable = More Cost – Additional copies, synchronization traffic, high cost hardware Not Scalable – Performance and management degrades as scale increases

6 Information Dispersal With the emergence of Broadband and modern microprocessors, Information Dispersal Algorithms (IDA’s) can be used to store the world’s data… – Inherently secure – Inherently private – Inherently reliable – Inherently long term Similar mathematical methods are the basis of digital mobile telephony and the Internet –Packet Switching, Reed-Solomon, Erasure Coding, Forward Error Correction, etc.

7 How Information Dispersal Works Information Dispersal Algorithms - Quick Mathematical Transformation 36 example characters = 36 total Bytes “Slices” are to data storage …what “packets” are to data communications. - Provide inherently reliable, private, secure and long-term storage 16 example slices = 58 total Bytes This Slicing example has a 60% Storage Overhead - Total bytes stored = 1.6X usable capacity

8 Dispersal versus Replication Width Threshold >16 11 >16 Storage Overhead 60% 33% 60% 33% 14% Access Choices million 214 million Typical Configurations Copies Parity No Yes No Yes Nines of Reliability >16 Storage Overhead 100% 167% 200% 300% Access Choices Nines of Reliability Slice Storage Copy Parity Copy Parity Copies + Parity Storage Dispersal Replication Bandwidth Overhead 60% 33% 60% 33% 14% Bandwidth Overhead 100% 200% Source data sizeStorage Overhead size Source data sizeStorage Overhead size

9 Data Storage with Information Dispersal Accesser dsNet Client Object iSCSI File Storage Overhead15-60%Maximum Delivery16 at once Bandwidth Needed15-60%Delivery choicesthousands

10 Data Retrieval with Information Dispersal Accesser dsNet Client iSCSI File Storage Overhead15-60%Maximum Delivery16 at once Bandwidth Needed15-60%Delivery choicesthousands X X X X Object

11 4 TB “Raw” capacity per 1U server Store, return and delete slices Unlimited vaults (similar to LUNs) per dsNet Deployable in a single rack or geographically distributed around the world Slicestor Slices-disperses-retrieves data to/from Slicestors Provides standard storage interfaces Ideal for digital content loading Deployable in redundant configurations Product – Appliance Components Accesser dsNet Client Dispersed Storage client software Dispersed Storage server Dispersed Storage router Disperses and retrieves data to/from slice servers Approximately 3 MB of Java code Ideal for content distribution

Exabytes of Data Storage Require a Paradigm Shift Scenario 1: Exbibyte (~1,000,000 Terabytes) Usable Storage, 10 nines of Reliability Replication: Total Raw Storage ~ 3.75 EiB Replication:~300% Storage overhead Additional costs for replication SW 1 EiB Usable 1 EiB Replicated 25% Parity 25% Parity 25% Parity Dispersal: Total Raw Storage ~ 1.33 EiB 33% Dispersal Overhead 1 EiB Usable Dispersal:~33% storage overhead built in multi-site availability Traditional Storage Dispersed Storage

13 dsNet – Standard Interfaces iSCSI: Acts like a hard drive -Works with any OS or file system Java client enables any device to access a dsNet: - Media players, phones, set top box, security cameras, sensors, etc. WebDAV: Acts like a URL -Works with any browser

Complete Storage Architecture 14 Vault Structure Interface Layer Protocol Layer ObjectBlock Simple Object SCSI File System Named Object Web Service iSCSI dsFTP WebDAV Java SDK REST API Network Client Layer

15 Project Overview An Open Source Project with Commercial Backing Dispersed Storage – an Open Source Project Hosted at Includes the complete protocol and algorithms Incorporates and/or enhances additional open source software –Bouncy Castle – Cryptography –JSAP – Java Simple Argument Parser –Bzip2 – Data Compressor –Apache Commons – Logging, Statistics, basic Internet protocols –JUnit – Testing Framework –Log4j – Logging Utility –MINA -- Network Application Framework –SLF4J – Simple Logging Façade for Java –SVNKit – Java Subversion library –Wrapper – Java Service Wrapper –ws-commons – Webservices Common Utilities –jSCSI – iSCSI Initiator

16 Open Source Complements Commercial Cleversafe Dispersed Storage Interoperability Protocols - Standards - Open Source software Products - Integrated hw/sw Appliances - Customized OS - Additional hardware features - Performance Services - Training - Certification - Support Additional Capabilities - Management - Reporting Open Source Commercial Contribute to standards efforts Commercial Capabilities Internet Equipment Providers

Thank You