Serverless Network File Systems. Network File Systems Allow sharing among independent file systems in a transparent manner Mounting a remote directory.

Slides:



Advertisements
Similar presentations
Multiple Processor Systems
Advertisements

RAID (Redundant Arrays of Independent Disks). Disk organization technique that manages a large number of disks, providing a view of a single disk of High.
G O O G L E F I L E S Y S T E M 陳 仕融 黃 振凱 林 佑恩 Z 1.
The Zebra Striped Network Filesystem. Approach Increase throughput, reliability by striping file data across multiple servers Data from each client is.
The Zebra Striped Network File System Presentation by Joseph Thompson.
11-May-15CSE 542: Operating Systems1 File system trace papers The Zebra striped network file system. Hartman, J. H. and Ousterhout, J. K. SOSP '93. (ACM.
Distributed Storage March 12, Distributed Storage What is Distributed Storage?  Simple answer: Storage that can be shared throughout a network.
CS-550: Distributed File Systems [SiS]1 Resource Management in Distributed Systems: Distributed File Systems.
U NIVERSITY OF M ASSACHUSETTS, A MHERST Department of Computer Science Emery Berger University of Massachusetts Amherst Operating Systems CMPSCI 377 Lecture.
Network Operating Systems Users are aware of multiplicity of machines. Access to resources of various machines is done explicitly by: –Logging into the.
Web Caching Schemes1 A Survey of Web Caching Schemes for the Internet Jia Wang.
G Robert Grimm New York University Sprite LFS or Let’s Log Everything.
Other File Systems: LFS and NFS. 2 Log-Structured File Systems The trend: CPUs are faster, RAM & caches are bigger –So, a lot of reads do not require.
Sinfonia: A New Paradigm for Building Scalable Distributed Systems Marcos K. Aguilera, Arif Merchant, Mehul Shah, Alistair Veitch, Christonos Karamanolis.
Cse Feb-001 CSE 451 Section February 24, 2000 Project 3 – VM.
Large Scale Sharing GFS and PAST Mahesh Balakrishnan.
G Robert Grimm New York University Sprite LFS or Let’s Log Everything.
16: Distributed Systems1 DISTRIBUTED SYSTEM STRUCTURES NETWORK OPERATING SYSTEMS The users are aware of the physical structure of the network. Each site.
PRASHANTHI NARAYAN NETTEM.
Microsoft Load Balancing and Clustering. Outline Introduction Load balancing Clustering.
Case Study - GFS.
RAID-x: A New Distributed Disk Array for I/O-Centric Cluster Computing Kai Hwang, Hai Jin, and Roy Ho.
Distributed File Systems Sarah Diesburg Operating Systems CS 3430.
Network File Systems Victoria Krafft CS /4/05.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google∗
PETAL: DISTRIBUTED VIRTUAL DISKS E. K. Lee C. A. Thekkath DEC SRC.
Presented by: Alvaro Llanos E.  Motivation and Overview  Frangipani Architecture overview  Similar DFS  PETAL: Distributed virtual disks ◦ Overview.
Distributed File Systems Concepts & Overview. Goals and Criteria Goal: present to a user a coherent, efficient, and manageable system for long-term data.
Distributed Data Stores – Facebook Presented by Ben Gooding University of Arkansas – April 21, 2015.
1 The Google File System Reporter: You-Wei Zhang.
CSC 456 Operating Systems Seminar Presentation (11/13/2012) Leon Weingard, Liang Xin The Google File System.
Module 12: Designing High Availability in Windows Server ® 2008.
N-Tier Client/Server Architectures Chapter 4 Server - RAID Copyright 2002, Dr. Ken Hoganson All rights reserved. OS Kernel Concept RAID – Redundant Array.
CH2 System models.
Distributed File Systems
CSE 451: Operating Systems Section 10 Project 3 wrap-up, final exam review.
Scalable Web Server on Heterogeneous Cluster CHEN Ge.
Distributed File Systems Overview  A file system is an abstract data type – an abstraction of a storage device.  A distributed file system is available.
Distributed File System By Manshu Zhang. Outline Basic Concepts Current project Hadoop Distributed File System Future work Reference.
Properties of Layouts Single failure correcting: no two units of same stripe are mapped to same disk –Enables recovery from single disk crash Distributed.
Introduction to DFS. Distributed File Systems A file system whose clients, servers and storage devices are dispersed among the machines of a distributed.
Presenters: Rezan Amiri Sahar Delroshan
Serverless Network File Systems Overview by Joseph Thompson.
Supporting Multi-Processors Bernard Wong February 17, 2003.
Fast Crash Recovery in RAMCloud. Motivation The role of DRAM has been increasing – Facebook used 150TB of DRAM For 200TB of disk storage However, there.
CS 153 Design of Operating Systems Spring 2015 Lecture 22: File system optimizations.
Presented By: Samreen Tahir Coda is a network file system and a descendent of the Andrew File System 2. It was designed to be: Highly Highly secure Available.
Computer Science Lecture 19, page 1 CS677: Distributed OS Last Class: Fault tolerance Reliable communication –One-one communication –One-many communication.
Distributed File Systems Architecture – 11.1 Processes – 11.2 Communication – 11.3 Naming – 11.4.
CS 6401 Overlay Networks Outline Overlay networks overview Routing overlays Resilient Overlay Networks Content Distribution Networks.
Distributed File Systems Questions answered in this lecture: Why are distributed file systems useful? What is difficult about distributed file systems?
GPFS: A Shared-Disk File System for Large Computing Clusters Frank Schmuck & Roger Haskin IBM Almaden Research Center.
1 CEG 2400 Fall 2012 Network Servers. 2 Network Servers Critical Network servers – Contain redundant components Power supplies Fans Memory CPU Hard Drives.
Distributed File System. Outline Basic Concepts Current project Hadoop Distributed File System Future work Reference.
Computer Science Lecture 19, page 1 CS677: Distributed OS Last Class: Fault tolerance Reliable communication –One-one communication –One-many communication.
Operating Systems Distributed-System Structures. Topics –Network-Operating Systems –Distributed-Operating Systems –Remote Services –Robustness –Design.
DISTRIBUTED FILE SYSTEM- ENHANCEMENT AND FURTHER DEVELOPMENT BY:- PALLAWI(10BIT0033)
Distributed File Systems
File System Implementation
Unit OS10: Fault Tolerance
Storage Virtualization
Outline Midterm results summary Distributed file systems – continued
Today: Coda, xFS Case Study: Coda File System
Overview Continuation from Monday (File system implementation)
Outline Announcements Lab2 Distributed File Systems 1/17/2019 COP5611.
Outline Review of Quiz #1 Distributed File Systems 4/20/2019 COP5611.
Ch 9 – Distributed Filesystem
Database System Architectures
Disk Scheduling The operating system is responsible for using hardware efficiently — for the disk drives, this means having a fast access time and disk.
Distributed Systems and Concurrency: Distributed Systems
Presentation transcript:

Serverless Network File Systems

Network File Systems Allow sharing among independent file systems in a transparent manner Mounting a remote directory in NFS Use remote procedure calls Traditional Network File Systems, like NFS, use a central server to provide the file system services. This work present an alternative, serverless network file system called xFS.

NFS Requirements / Metrics Performance / Speed Availability Scalability Fault Tolerance / Recovery

Limitations of Central Server Systems All read misses and disk writes go to the server – Performance Bottleneck Not scalable – Too many clients can hurt performance. Expensive to increase server hardware or add servers. Require server replication for high availability – increases cost and complexity. Also, latency to duplicate the data.

Serverless Network File Systems Increased Performance - distributes control processing and data storage among cooperating workstations. Scales easily to simplify system management. Fault tolerance through distributed RAID and Log structured file system. Migrates responsibility of failed components to other workstations.

Background RAID – Write portion of data to each disk. High performance (parallel accesses) Availability xFS uses RAID striping for files across a stripe group. Small writes hurt performance – must do parity update. Log-structured File System (LFS) Append-only file system. Leaves holes – need cleanup. Quick writes can be delayed to help RAID Helps recovery (checkpoints on disk)

Background (con’t) Multiprocessor Cache Consistency Statically divide physical memory evenly among processors. Each processor manages the cache consistency state for its own physical memory. xFS does this for files. The node storing the files keeps up with consistency. In xFS it is dynamic – files can be managed by different nodes.

Goals of xFS Provide a scalable way to subset storage servers into groups to provide efficient storage. Scalable, distributed metadata and cache consistency management. Flexibility to dynamically reconfigure responsibilities after failures.

System entities Clients – want to access data in the system Storage Servers – store the system’s files Metadata Managers – hold cache consistency state and disk location metadata. Cleaners – clean up the LFS after writes Entities may lie on the same system or on different systems.

Serverless File Service “Anything, Anywhere” – all data and metadata can be located on and move to any node in the system. File access is faster because they are distributed across multiple workstations! How does the system locate the data? Key maps: manager map, imap, file directories, stripe group maps.

Manager Map Table indicating which machines manage which file indices File indices listed in the parent directory file. Globally Replicated Updated dynamically On machine failure or reconfiguration of file managers Can work as a load balancing mechanism. Not yet implemented, but a possibility

Imap Imaps are held by a file’s manager Maps a file’s index number to the disk address of the index node (inode). The index node gives the file offset and pointers to each data block. Similar to standard OS implementation

RAID Stripe Groups Better to stripe files over a group of servers instead of all servers in the system. Improves availability – Each group stores its own parity. Allows recovery from multiple failures. Stripe Group Map – tells which nodes are a member of the group. Must reference this map before reading or writing data to the file system.

Cache Consistency Token-based scheme Client must request and acquire write ownership from the file’s manager. Manager invalidates other cached copies

Cache Consistency (con’t) Client keeps write ownership until another client requests it. It then must flush the changes to the disk. xFS guarantees that the up-to-date copy is given to the node requesting the data. Traditional network file systems do not always guarantee this.

Management Distribution Policies xFS tries to assign files used by a client to a manager co-located on that machine. When a client creates a file, xFS assigns the manager on that machine to the file. Improves locality Reduces network hops to satisfy requests - 40%

Reconfiguration Not yet implemented in this version… When system detects configuration change, a global consensus algorithm is envoked. Leader is chosen to run the algorithm given a list of active nodes. Generates a new manager map and distributes it across the nodes.

Security in xFS Only appropriate in a restricted environment Machines cooperating over a fast network Must trust one another’s kernels to enforce security