Coding for Modern Distributed Storage Systems: Part 1. Locally Repairable Codes Parikshit Gopalan Windows Azure Storage, Microsoft.

Slides:

Advertisements

Similar presentations

A Case for Redundant Arrays Of Inexpensive Disks Paper By David A Patterson Garth Gibson Randy H Katz University of California Berkeley.

Advertisements

Redundant Array of Independent Disks (RAID) Striping of data across multiple media for expansion, performance and reliability.

Distributed Storage Systems Vinodh Venkatesan IBM Zurich Research Lab / EPFL.

Analysis and Construction of Functional Regenerating Codes with Uncoded Repair for Distributed Storage Systems Yuchong Hu, Patrick P. C. Lee, Kenneth.

current hadoop architecture

Alex Dimakis based on collaborations with Dimitris Papailiopoulos Arash Saber Tehrani USC Network Coding for Distributed Storage.

CSCE430/830 Computer Architecture

1 NCFS: On the Practicality and Extensibility of a Network-Coding-Based Distributed File System Yuchong Hu 1, Chiu-Man Yu 2, Yan-Kit Li 2 Patrick P. C.

 RAID stands for Redundant Array of Independent Disks  A system of arranging multiple disks for redundancy (or performance)  Term first coined in 1987.

BASIC Regenerating Codes for Distributed Storage Systems Kenneth Shum (Joint work with Minghua Chen, Hanxu Hou and Hui Li)

Note: is very restrictive Would like more flexibility in determining the length If we shorten the length of the message.

Coding and Algorithms for Memories Lecture 12 1.

Simple Regenerating Codes: Network Coding for Cloud Storage Dimitris S. Papailiopoulos, Jianqiang Luo, Alexandros G. Dimakis, Cheng Huang, and Jin Li University.

Yuchong Hu1, Henry C. H. Chen1, Patrick P. C. Lee1, Yang Tang2

RAID- Redundant Array of Inexpensive Drives. Purpose Provide faster data access and larger storage Provide data redundancy.

RAID Redundant Arrays of Inexpensive Disks –Using lots of disk drives improves: Performance Reliability –Alternative: Specialized, high-performance hardware.

Chapter 5: Server Hardware and Availability. Hardware Reliability and LAN The more reliable a component, the more expensive it is. Server hardware is.

Lecture 36: Chapter 6 Today’s topic –RAID 1. RAID Redundant Array of Inexpensive (Independent) Disks –Use multiple smaller disks (c.f. one large disk)

Sean Traber CS-147 Fall  7.9 RAID  RAID Level 0  RAID Level 1  RAID Level 2  RAID Level 3  RAID Level 4 

REDUNDANT ARRAY OF INEXPENSIVE DISCS RAID. What is RAID ? RAID is an acronym for Redundant Array of Independent Drives (or Disks), also known as Redundant.

An Upper Bound on Locally Recoverable Codes Viveck R. Cadambe (MIT) Arya Mazumdar (University of Minnesota)

Beyond the MDS Bound in Distributed Cloud Storage

Coding for Atomic Shared Memory Emulation Viveck R. Cadambe (MIT) Joint with Prof. Nancy Lynch (MIT), Prof. Muriel Médard (MIT) and Dr. Peter Musial (EMC)

Parikshit Gopalan Windows Azure Storage, Microsoft.

The Google File System. Why? Google has lots of data –Cannot fit in traditional file system –Spans hundreds (thousands) of servers connected to (tens.

Cooperative regenerating codes for distributed storage systems Kenneth Shum (Joint work with Yuchong Hu) 22nd July 2011.

Codes with local decoding procedures Sergey Yekhanin Microsoft Research.

Unit 3—Part A Computer Memory

Failures in the System  Two major components in a Node Applications System.

Network Coding for Distributed Storage Systems IEEE TRANSACTIONS ON INFORMATION THEORY, SEPTEMBER 2010 Alexandros G. Dimakis Brighten Godfrey Yunnan Wu.

1 Fault-Tolerant Computing Systems #2 Hardware Fault Tolerance Pattara Leelaprute Computer Engineering Department Kasetsart University

LAN / WAN Business Proposal. What is a LAN or WAN? A LAN is a Local Area Network it usually connects all computers in one building or several building.

1 Recitation 8 Disk & File System. 2 Disk Scheduling Disks are at least four orders of magnitude slower than main memory –The performance of disk I/O.

RAID: High-Performance, Reliable Secondary Storage Mei Qing & Chaoxia Liao Nov. 20, 2003.

©2001 Pål HalvorsenINFOCOM 2001, Anchorage, April 2001 Integrated Error Management in MoD Services Pål Halvorsen, Thomas Plagemann, and Vera Goebel University.

Repairable Fountain Codes Megasthenis Asteris, Alexandros G. Dimakis IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 32, NO. 5, MAY /5/221.

Redundant Array of Inexpensive Disks aka Redundant Array of Independent Disks (RAID) Modified from CCT slides.

CSI-09 COMMUNICATION TECHNOLOGY FAULT TOLERANCE AUTHOR: V.V. SUBRAHMANYAM.

Computing in the RAIN: A Reliable Array of Independent Nodes Group A3 Ka Hou Wong Jahanzeb Faizan Jonathan Sippel.

4.1 © 2004 Pearson Education, Inc. Exam Managing and Maintaining a Microsoft® Windows® Server 2003 Environment Lesson 4: Organizing a Disk for Data.

1 Network Coding and its Applications in Communication Networks Alex Sprintson Computer Engineering Group Department of Electrical and Computer Engineering.

"1"1 Introduction to Managing Data " Describe problems associated with managing large numbers of disks " List requirements for easily managing large amounts.

A Cost-based Heterogeneous Recovery Scheme for Distributed Storage Systems with RAID-6 Codes Yunfeng Zhu 1, Patrick P. C. Lee 2, Liping Xiang 1, Yinlong.

Cooperative Recovery of Distributed Storage Systems from Multiple Losses with Network Coding Yuchong Hu, Yinlong Xu, Xiaozhao Wang, Cheng Zhan and Pei.

Chapter 12 – Mass Storage Structures (Pgs )

Coding and Algorithms for Memories Lecture 14 1.

Davie 5/18/2010.  Thursday, May 20 5:30pm  Ursa Minor  Co-sponsored with CSS  Guest Speakers  Dr. Craig Rich – TBA  James Schneider – Cal Poly.

Page 1 File Systems and Disk Management Lecture 5 Hassan Shuja 10/19/2004.

RAID Systems Ver.2.0 Jan 09, 2005 Syam. RAID Primer Redundant Array of Inexpensive Disks random, real-time, redundant, array, assembly, interconnected,

Exact Regenerating Codes on Hierarchical Codes Ernst Biersack Eurecom France Joint work and Zhen Huang.

20/10/ Cooperative Recovery of Distributed Storage Systems from Multiple Losses with Network Coding Yuchong Hu Institute of Network Coding Please.

A Fast Repair Code Based on Regular Graphs for Distributed Storage Systems Yan Wang, East China Jiao Tong University Xin Wang, Fudan University 1 12/11/2013.

Coding and Algorithms for Memories Lecture 13 1.

A Mechanism for Communication- Efficient Broadcast Encryption over Wireless Ad Hoc Networks Johns Hopkins University Department of Computer Science Reza.

Chapter 14: Mass-Storage Systems Disk Structure. Disk Scheduling. RAID.

Secret Sharing in Distributed Storage Systems Illinois Institute of Technology Nexus of Information and Computation Theories Paris, Feb 2016 Salim El Rouayheb.

Hands-On Microsoft Windows Server 2008 Chapter 7 Configuring and Managing Data Storage.

Seminar On Rain Technology

RAID Technology By: Adarsha A,S 1BY08A03. Overview What is RAID Technology? What is RAID Technology? History of RAID History of RAID Techniques/Methods.

A Tale of Two Erasure Codes in HDFS

IERG6120 Lecture 22 Kenneth Shum Dec 2016.

RAID Redundant Arrays of Independent Disks

Vladimir Stojanovic & Nicholas Weaver

Principles of Information Technology

Section 7 Erasure Coding Overview

RAID RAID Mukesh N Tekwani

Maximally Recoverable Local Reconstruction Codes

RAID RAID Mukesh N Tekwani April 23, 2019

CS 295: Modern Systems Organizing Storage Devices

Seminar on Enterprise Software

Presentation transcript:

Coding for Modern Distributed Storage Systems: Part 1. Locally Repairable Codes Parikshit Gopalan Windows Azure Storage, Microsoft.

The data deluge … Problem Statement: We generate insane amounts of digital data. We expect it to be stored reliably and accessible anytime, anywhere. And for free!  Total data in the cloud is of the order of few hundred Exabytes  A terabyte hard drive costs $100.  Even storing raw data costs hundreds of millions. Hardware is no longer cheap. Currently, data centers consume up to 3 percent of all global electricity production while producing 200 million metric tons of carbon dioxide.

Data Storage: the basics Goal: Tolerate one disk failure.  Duplication 2X overhead. Quick recovery.  Simple XOR [RAID5] Treat each disk as a bit vector. 1.2X overhead. Slower recovery.

Data Storage: the basics Goal: Tolerate two disk failures.  Triplication 3X overhead. Quick recovery.  [6,4] Reed Solomon Code [RAID6] 1.5X overhead. Slower recovery. Need a larger field: each disk is a byte-vector.

Data storage: the basics

Degraded Reads Typical failure scenario: a single disk fails or is prohibitively slow.

Degraded Reads Typical failure scenario: a single disk fails or is prohibitively slow.

Disk Failure Typical failure scenario: a single disk fails and needs to be replaced.

Can we do better? Regenerating Codes [Dimakis-Godfrey-Wu-Wainwright-Ramachandran’10] Metric: Network bandwidth. Optimize the amount of data communicated to repair a single lost node. Locally Repairable Codes [G.-Huang-Simitci-Yekhanin’12] Metric: Number of disk reads. Optimize the number of disk reads needed to repair a single lost node.

Part 1 of this Tutorial: LRCs Part 1.1: Locality 1.Locality of codeword symbols. 2.Rate-distance-locality tradeoffs. Part 1.2: Reliability 1.Beyond minimum distance: Maximum recoverability. 2.Constructions of Maximally Recoverable LRCs.

Locality [Chen-Huang-Li’07, Oggier-Datta’11, G.-Simitci-Huang-Yekhanin’12, Papailiopoulos-Luo-Dimakis-Huang-Li’12]

Locally Decodable/Testable Codes

Codes with data locality

Codes with all-symbol locality

Rate-distance-locality tradeoffs

Explicit codes with all-symbol locality.

Rate-distance-locality tradeoffs

Generalizations  Non-linear codes [Papailiopoulos-Dimakis, Forbes-Yekhanin].  Vector codes [Papailoupoulos-Dimakis, Silberstein-Rawat-Koyluoglu-Vishwanath, Kamath- Prakash-Lalitha-Kumar]  Codes over bounded alphabets [Cadambe-Mazumdar]  Codes with short local MDS codes [Prakash-Lalitha-Kamath-Kumar, Silberstein-Rawat-Koyluoglu-Vishwanath]  Codes with local Regeneration [Silberstein-Rawat-Koyluoglu-Vishwanath, Kamath-Prakash-Lalitha-Kumar…]

Towards locally decodable codes