Data Management Reading: Chapter 5: Data-Intensive Computing And A Network-Aware Distributed Storage Cache for Data Intensive Environments.

Slides:



Advertisements
Similar presentations
Redundant Array of Independent Disks (RAID) Striping of data across multiple media for expansion, performance and reliability.
Advertisements

What is RAID Redundant Array of Independent Disks.
1 Lecture 18: RAID n I/O bottleneck n JBOD and SLED n striping and mirroring n classic RAID levels: 1 – 5 n additional RAID levels: 6, 0+1, 10 n RAID usage.
IT 344: Operating Systems Winter 2007 Module 18 Redundant Arrays of Inexpensive Disks (RAID) Chia-Chi Teng CTB 265.
Faculty of Information Technology Department of Computer Science Computer Organization Chapter 7 External Memory Mohammad Sharaf.
CSE 451: Operating Systems Spring 2012 Module 20 Redundant Arrays of Inexpensive Disks (RAID) Ed Lazowska Allen Center 570 ©
RAID (Redundant Arrays of Independent Disks). Disk organization technique that manages a large number of disks, providing a view of a single disk of High.
RAID Oh yes Whats RAID? Redundant Array (of) Independent Disks. A scheme involving multiple disks which replicates data across multiple drives. Methods.
Magnetic Disk Magnetic disks are the foundation of external memory on virtually all computer systems. A disk is a circular platter constructed of.
RAID Redundant Array of Independent Disks
CS 6560: Operating Systems Design
CSCE430/830 Computer Architecture
High Performance Computing Course Notes Grid Computing.
RAID- Redundant Array of Inexpensive Drives. Purpose Provide faster data access and larger storage Provide data redundancy.
Serverless Network File Systems. Network File Systems Allow sharing among independent file systems in a transparent manner Mounting a remote directory.
R.A.I.D. Copyright © 2005 by James Hug Redundant Array of Independent (or Inexpensive) Disks.
2P13 Week 11. A+ Guide to Managing and Maintaining your PC, 6e2 RAID Controllers Redundant Array of Independent (or Inexpensive) Disks Level 0 -- Striped.
CSE521: Introduction to Computer Architecture Mazin Yousif I/O Subsystem RAID (Redundant Array of Independent Disks)
REDUNDANT ARRAY OF INEXPENSIVE DISCS RAID. What is RAID ? RAID is an acronym for Redundant Array of Independent Drives (or Disks), also known as Redundant.
1 Storage (cont’d) Disk scheduling Reducing seek time (cont’d) Reducing rotational latency RAIDs.
1 Lecture 26: Storage Systems Topics: Storage Systems (Chapter 6), other innovations Final exam stats:  Highest: 95  Mean: 70, Median: 73  Toughest.
I/O Systems and Storage Systems May 22, 2000 Instructor: Gary Kimura.
RAID Systems CS Introduction to Operating Systems.
Secondary Storage Unit 013: Systems Architecture Workbook: Secondary Storage 1G.
1 Fault-Tolerant Computing Systems #2 Hardware Fault Tolerance Pattara Leelaprute Computer Engineering Department Kasetsart University
CS 352 : Computer Organization and Design University of Wisconsin-Eau Claire Dan Ernst Storage Systems.
CSE 321b Computer Organization (2) تنظيم الحاسب (2) 3 rd year, Computer Engineering Winter 2015 Lecture #4 Dr. Hazem Ibrahim Shehata Dept. of Computer.
Redundant Array of Independent Disks
L/O/G/O External Memory Chapter 3 (C) CS.216 Computer Architecture and Organization.
N-Tier Client/Server Architectures Chapter 4 Server - RAID Copyright 2002, Dr. Ken Hoganson All rights reserved. OS Kernel Concept RAID – Redundant Array.
I/O – Chapter 8 Introduction Disk Storage and Dependability – 8.2 Buses and other connectors – 8.4 I/O performance measures – 8.6.
Lecture 9 of Advanced Databases Storage and File Structure (Part II) Instructor: Mr.Ahmed Al Astal.
Redundant Array of Inexpensive Disks aka Redundant Array of Independent Disks (RAID) Modified from CCT slides.
ARGONNE  CHICAGO Ian Foster Discussion Points l Maintaining the right balance between research and development l Maintaining focus vs. accepting broader.
File and Object Replication in Data Grids Chin-Yi Tsai.
RAID SECTION (2.3.5) ASHLEY BAILEY SEYEDFARAZ YASROBI GOKUL SHANKAR.
Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.
Redundant Array of Independent Disks.  Many systems today need to store many terabytes of data.  Don’t want to use single, large disk  too expensive.
"1"1 Introduction to Managing Data " Describe problems associated with managing large numbers of disks " List requirements for easily managing large amounts.
CS 153 Design of Operating Systems Spring 2015 Lecture 22: File system optimizations.
The concept of RAID in Databases By Junaid Ali Siddiqui.
Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.
1 CEG 2400 Fall 2012 Network Servers. 2 Network Servers Critical Network servers – Contain redundant components Power supplies Fans Memory CPU Hard Drives.
Hands-On Microsoft Windows Server 2008 Chapter 7 Configuring and Managing Data Storage.
RAID Technology By: Adarsha A,S 1BY08A03. Overview What is RAID Technology? What is RAID Technology? History of RAID History of RAID Techniques/Methods.
RAID TECHNOLOGY RASHMI ACHARYA CSE(A) RG NO
Silberschatz, Galvin and Gagne ©2013 Operating System Concepts – 9 th Edition Chapter 10: Mass-Storage Systems.
CS Introduction to Operating Systems
RAID.
CMPE Database Systems Workshop June 16 Class Meeting
Multiple Platters.
RAID, Programmed I/O, Interrupt Driven I/O, DMA, Operating System
Operating System I/O System Monday, August 11, 2008.
Storage Virtualization
CSE 451: Operating Systems Spring 2006 Module 18 Redundant Arrays of Inexpensive Disks (RAID) John Zahorjan Allen Center.
RAID RAID Mukesh N Tekwani
Chapter 17: Database System Architectures
CSE 451: Operating Systems Autumn 2010 Module 19 Redundant Arrays of Inexpensive Disks (RAID) Ed Lazowska Allen Center 570.
CSE 451: Operating Systems Winter 2009 Module 13 Redundant Arrays of Inexpensive Disks (RAID) and OS structure Mark Zbikowski Gary Kimura 1.
O.S Lecture 14 File Management.
TECHNICAL SEMINAR PRESENTATION
UNIT IV RAID.
Mark Zbikowski and Gary Kimura
CSE 451: Operating Systems Winter 2012 Redundant Arrays of Inexpensive Disks (RAID) and OS structure Mark Zbikowski Gary Kimura 1.
CSE 451: Operating Systems Winter 2007 Module 18 Redundant Arrays of Inexpensive Disks (RAID) Ed Lazowska Allen Center 570.
CSE 451: Operating Systems Autumn 2009 Module 19 Redundant Arrays of Inexpensive Disks (RAID) Ed Lazowska Allen Center 570.
RAID RAID Mukesh N Tekwani April 23, 2019
Database System Architectures
IT 344: Operating Systems Winter 2007 Module 18 Redundant Arrays of Inexpensive Disks (RAID) Chia-Chi Teng CTB
Improving performance
Presentation transcript:

Data Management Reading: Chapter 5: Data-Intensive Computing And A Network-Aware Distributed Storage Cache for Data Intensive Environments

What is Data Management? It depends… l Storage systems Disk arrays Network caches (e.g., DPSS) Hierarchical storage systems (e.g., HPSS) l Efficient data transport mechanisms Striped Parallel Secure Reliable Third-party transfers

What is data management? (cont.) l Replication management Associate files into collections Mechanisms for reliably copying collections, propagating updates to collections, selecting among replicas l Metadata management Associate attributes that describe data Select data based on attributes l Publishing and curation of data Official versions of important collections Digital libraries

Outline for Today l Examples of data-intensive applications l Storage systems: Disk arrays High-Performance Network Caches (DPSS) Hierarchical Storage Systems (Chris: HPSS) l Next two lectures: gridFTP Globus replica management Metadata systems Curation

Data-Intensive Applications: Physics l CERN Large Hadron Collider l Several terabytes of data per year Starting in 2005 Continuing 15 to 20 years Replication scenario: Copy of everything at CERN (Tier 0) Subsets at national centers (Tier 1) Smaller regional centers (Tier 2) Individual researchers will have copies

GriPhyN Overview ( l 5-year, $12.5M NSF ITR proposal to realize the concept of virtual data, via: l Key research areas: Virtual data technologies (information models, management of virtual data software, etc.) Request planning and scheduling (including policy representation and enforcement) Task execution (including agent computing, fault management, etc.) l Development of Virtual Data Toolkit (VDT) l Four Applications: ATLAS, CMS, LIGO, SDSS

GriPhyN Participants l Computer Science U.Chicago, USC/ISI, UW-Madison, UCSD, UCB, Indiana, Northwestern, Florida l Toolkit Development U.Chicago, USC/ISI, UW-Madison, Caltech l Applications ATLAS (Indiana), CMS (Caltech), LIGO (UW-Milwaukee, UT-B, Caltech), SDSS (JHU) l Unfunded collaborators UIC (STAR-TAP), ANL, LBNL, Harvard, U.Penn

The Petascale Virtual Data Grid (PVDG) Model l Data suppliers publish data to the Grid l Users request raw or derived data from Grid, without needing to know Where data is located Whether data is stored or computed l User can easily determine What it will cost to obtain data Quality of derived data l PVDG serves requests efficiently, subject to global and local policy constraints

PVDG Scenario User requests may be satisfied via a combination of data access and computation at local, regional, and central sites

Other Application Scenarios l Climate community Terabyte-scale climate model datasets: Collecting measurements Simulation results Must support sharing, remote access to and analysis of datasets l Distance visualization Remote navigation through large datasets, with local and/or remote computing

Storage Systems: Disk Arrays l What is a disk array? Collection of disks l Advantages: Higher capacity Many small, inexpensive disks Higher throughput Higher bandwidth (Mbytes/sec) on large transfers Higher I/O rate (transactions/sec) on small transfers

Trends in Magnetic Disks l Capacity increases: 60% per year l Cost falling at similar rate ($/MB or $/GB) l Evolving to smaller physical sizes 14in 5.25in 3.5in 2.5in 1.0in … ? l Put lots of small disks together l Problem: RELIABILITY Reliability of N disks = Reliability of 1 disk divided by N

Key Concepts in Disk Arrays Striping for High Performance Interleave data from single file across multiple disks Fine-grained interleaving: every file spread across all disks any access involves all disks Course-grained interleaving: interleave in large blocks small accesses may be satisfied by a single disk

Key Concepts in Disk Arrays Redundancy Maintain extra information in disk array Duplication Parity Reed-Solomon error correction codes Others When a disk fails: use redundancy information to reconstruct data on failed disk

RAID Levels l Defined by combinations of striping & redundancy l RAID Level 1: Mirroring or Shadowing Maintain a complete copy of each disk Very reliable High cost: twice the number of disks Great performance: on a read, may go to disk with faster access time l RAID Level 2: Memory Style Error Detection and Correction Not really implemented in practice Based on DRAM-style Hamming codes In disk systems, dont need detection Use less expensive correction schemes

RAID Levels (cont.) l RAID Level 3: Fine-grained Interleaving and Parity Many commercial RAIDs Calclate parity bit-wise across disks in the array (using exclusive-OR logic) Maintain a separate parity disk; update on write operations When a disk fails, use other data disk and parity disk to reconstruct data on lost disk Fine-grained interleaving: all disks involved in any access to the array

RAID Levels (cont.) l RAID Level 4: Large Block Interleaving and Parity Similar to level 3, but interleave on larger blocks Small accesses may be satisfied by a single disk Supports higher rate of small I/Os Parity disk may become a bottleneck with multiple concurrent I/Os l RAID Level 5: Large Block Interleaving and Distributed Parity Similar to level 4 Distributes parity blocks throughout all disks in array

RAID Levels (cont.) l RAID Level 6: Reed-Solomon Error Correction Codes Protection against two disk failures l Disks getting so cheap: consider massive storage systems composed entirely of disks No tape!!

DPSS: Distributed Parallel Storage System l Produced by Lawrence Berkeley National Labs l Cache: provides storage that is Faster than typical local disk Temporary l Virtual disk: appears to be single large, random- access, block-oriented I/O device l Isolates application from tertiary storage system: Acts as large buffer between slow tertiary storage and high-performance network connections Impedance matching

Features of DPSS l Components: DPSS block servers Typically low-cost workstations Each with several disk controllers, several disks per controller DPSS mater process Data requests sent from client to master process Determines which DPSS block server stores the requested blocks Forwards request to that block server Note: servers can be anywhere on network (a distributed cache)

Features of DPSS (cont.) l Client API library Supports variety of I/O semantics dpssOpen(), dpssRead(), dpssWrite(), dpssLSeek(), dpssClose() l Application controls data layout in cache For typical applications that read sequentially: stripe blocks of data across servers in round-robin fashion l DPSS client library is multi-threaded Number of client threads is equal to number of DPSS servers: client speed scales with server speed

Features of DPSS (cont.) l Optimized for relatively small number of large files Several thousand files Greater than 50 MB l DPSS blocks are available as soon as they are placed in cache Good for staging larges files to/from tertiary storage Dont have to wait for large transfer to complete l Dynamically reconfigurable Add or remove servers or disks on the fly

Features of DPSS (cont.) l Agent-based performance monitoring system l Client library automatically sets TCP buffer size to optimal value Uses information published by monitoring system l Load balancing Supports replication of files on multiple servers DPSS master uses status information stored in LDAP directory to select a replica that will give fastest response

Hierarchical Storage System l Fast, disk cache in front of larger, slower storage l Works on same principle as other hierarchies: Level-1 and Level-2 caches: minimize off-chip memory accesses Virtual memory systems:minimize page faults to disk l Goal: Keep popular material in faster storage Keep most of material on cheaper, slower storage Locality: 10% of material gets 90% of accesses l Problem with tertiary storage (especially tape): Very slow Tape seek times can be a minute or more…