Presentation is loading. Please wait.

Presentation is loading. Please wait.

Limitless Storage, Boundless Opportunities Technology Overview – January 2009.

Similar presentations


Presentation on theme: "Limitless Storage, Boundless Opportunities Technology Overview – January 2009."— Presentation transcript:

1 Limitless Storage, Boundless Opportunities Technology Overview – January 2009

2 Cleversafe Mission Enable the world to confidently store and distribute limitless data Commercial Products Providing Products and Service to companies who are building and operating Dispersed Storage Networks Dispersed Storage - Open Source Project Creating a genuine open standard and a broad technical community 2

3 3 Data Storage Growth Traditional DataAdditional, New Data Digital Content - 85% of all data by 2012 - Growing 10x every 4 years Images – 500KB per picture Audio – 5,000 KB per song Video – 5,000,000 KB per movie Documents Character & numerical databases + Source: IDC

4 4 Current High Availability Scenario 300% Disk Storage Overhead + Tape Backup -Total bytes stored = 4x usable capacity 200% Bandwidth Overhead -Each node supports full operational requirement -Total bandwidth required = 3x operational requirement Higher Cost - More Power - More Management - More Space - More Equipment More Security Risks ParityA3 RAID3 Controller Server1 @ Location 1 A2 A1 The quick brown fox jumps over the lazy brown dog. 11010010 00110010 Internet Connection Internet Connection Internet Connection ParityA3 RAID3 Controller Server2 @ Location 2 A2 A1 The quick brown fox jumps over the lazy brown dog. 11010010 00110010 ParityA3 RAID3 Controller Server3 @ Location 3 A2 A1 The quick brown fox jumps over the lazy brown dog. 11010010 00110010

5 5 Digital Data Storage - An Antiquated Approach Currently Data Storage = Data Copies Not Secure – 200 major announced security breaches since 2004 Not Private – Data copies are… data copies Not Long Term – Tied to hardware which doesn’t last over 5 years More Reliable = More Cost – Additional copies, synchronization traffic, high cost hardware Not Scalable – Performance and management degrades as scale increases

6 6 Information Dispersal With the emergence of Broadband and modern microprocessors, Information Dispersal Algorithms (IDA’s) can be used to store the world’s data… – Inherently secure – Inherently private – Inherently reliable – Inherently long term Similar mathematical methods are the basis of digital mobile telephony and the Internet –Packet Switching, Reed-Solomon, Erasure Coding, Forward Error Correction, etc.

7 7 How Information Dispersal Works Information Dispersal Algorithms - Quick Mathematical Transformation 36 example characters = 36 total Bytes “Slices” are to data storage …what “packets” are to data communications. - Provide inherently reliable, private, secure and long-term storage 16 example slices = 58 total Bytes This Slicing example has a 60% Storage Overhead - Total bytes stored = 1.6X usable capacity

8 8 Dispersal versus Replication Width 8 16 32 64 Threshold 5 6 10 12 24 56 6 7 >16 11 >16 Storage Overhead 60% 33% 60% 33% 14% Access Choices 70 28 8008 1820 11 million 214 million Typical Configurations Copies 22332233 Parity No Yes No Yes Nines of Reliability 5 10 7 >16 Storage Overhead 100% 167% 200% 300% Access Choices 22332233 Nines of Reliability Slice Storage Copy Parity Copy Parity Copies + Parity Storage Dispersal Replication 12345678910111213141516 12345678910111213141516 12345678 58 1234567891027282930313211121314151617181920252621222324 Bandwidth Overhead 60% 33% 60% 33% 14% Bandwidth Overhead 100% 200% Source data sizeStorage Overhead size Source data sizeStorage Overhead size 432176

9 9 Data Storage with Information Dispersal Accesser dsNet Client Object iSCSI File Storage Overhead15-60%Maximum Delivery16 at once Bandwidth Needed15-60%Delivery choicesthousands

10 10 Data Retrieval with Information Dispersal Accesser dsNet Client iSCSI File Storage Overhead15-60%Maximum Delivery16 at once Bandwidth Needed15-60%Delivery choicesthousands X X X X Object

11 11 4 TB “Raw” capacity per 1U server Store, return and delete slices Unlimited vaults (similar to LUNs) per dsNet Deployable in a single rack or geographically distributed around the world Slicestor Slices-disperses-retrieves data to/from Slicestors Provides standard storage interfaces Ideal for digital content loading Deployable in redundant configurations Product – Appliance Components Accesser dsNet Client Dispersed Storage client software Dispersed Storage server Dispersed Storage router Disperses and retrieves data to/from slice servers Approximately 3 MB of Java code Ideal for content distribution

12 Exabytes of Data Storage Require a Paradigm Shift Scenario 1: Exbibyte (~1,000,000 Terabytes) Usable Storage, 10 nines of Reliability Replication: Total Raw Storage ~ 3.75 EiB Replication:~300% Storage overhead Additional costs for replication SW 1 EiB Usable 1 EiB Replicated 25% Parity 25% Parity 25% Parity Dispersal: Total Raw Storage ~ 1.33 EiB 33% Dispersal Overhead 1 EiB Usable Dispersal:~33% storage overhead built in multi-site availability Traditional Storage Dispersed Storage

13 13 dsNet – Standard Interfaces iSCSI: Acts like a hard drive -Works with any OS or file system Java client enables any device to access a dsNet: - Media players, phones, set top box, security cameras, sensors, etc. WebDAV: Acts like a URL -Works with any browser

14 Complete Storage Architecture 14 Vault Structure Interface Layer Protocol Layer ObjectBlock Simple Object SCSI File System Named Object Web Service iSCSI dsFTP WebDAV Java SDK REST API Network Client Layer

15 15 Project Overview An Open Source Project with Commercial Backing Dispersed Storage – an Open Source Project Hosted at www.cleversafe.orgwww.cleversafe.org Includes the complete protocol and algorithms Incorporates and/or enhances additional open source software –Bouncy Castle – Cryptography –JSAP – Java Simple Argument Parser –Bzip2 – Data Compressor –Apache Commons – Logging, Statistics, basic Internet protocols –JUnit – Testing Framework –Log4j – Logging Utility –MINA -- Network Application Framework –SLF4J – Simple Logging Façade for Java –SVNKit – Java Subversion library –Wrapper – Java Service Wrapper –ws-commons – Webservices Common Utilities –jSCSI – iSCSI Initiator

16 16 Open Source Complements Commercial Cleversafe Dispersed Storage Interoperability Protocols - Standards - Open Source software Products - Integrated hw/sw Appliances - Customized OS - Additional hardware features - Performance Services - Training - Certification - Support Additional Capabilities - Management - Reporting Open Source Commercial Contribute to standards efforts Commercial Capabilities Internet Equipment Providers

17 Thank You


Download ppt "Limitless Storage, Boundless Opportunities Technology Overview – January 2009."

Similar presentations


Ads by Google