Intro to DFS assignment. Announcements DFS (PA4) Part 1 will be out over the weekend Part 1 Due 10/30 11:59 pm Part 2 12/7 11:59 pm Part 3 12/14 11:59.

Slides:



Advertisements
Similar presentations
Message Passing Vs Distributed Objects
Advertisements

CS140 Review Session Project 4 – File Systems Samir Selman 02/27/09cs140 Review Session Due Thursday March,11.
CSCC69: Operating Systems
Remote Procedure Call (RPC)
Remote Procedure Call Design issues Implementation RPC programming
The Zebra Striped Network Filesystem. Approach Increase throughput, reliability by striping file data across multiple servers Data from each client is.
© 2012 Whamcloud, Inc. Distributed Namespace Status Phase I - Remote Directories Wang Di Whamcloud, Inc.
Tam Vu Remote Procedure Call CISC 879 – Spring 03 Tam Vu March 06, 03.
Distributed systems Programming with threads. Reviews on OS concepts Each process occupies a single address space.
Multiple Processor Systems
FONG CHAN SING (143334) WONG YEW JOON (143388). JAVA RMI is a distributive system programming interface introduced in JDK 1.1. A library that allows an.
L-7 RPC 1. Last Lecture Important Lessons - Naming Naming is a powerful tool in system design  A layer of indirection can solve many problems Name+context.
Distributed systems Programming with threads. Reviews on OS concepts Each process occupies a single address space.
Remote Procedure Call in SR Programming Language By Tze-Kin Tsang 3/20/2000.
6/24/2015B.RamamurthyPage 1 File System B. Ramamurthy.
Jan 28, 2003CS475: Internetworking with UNIX TCP/IP1 XDR, RPC, NFS Internetworking with UNIX TCP/IP Winter
CHUBBY and PAXOS Sergio Bernales 1Dennis Kafura – CS5204 – Operating Systems.
Netprog RPC Overview1 Distributed Program Design n Communication-Oriented Design –Design protocol first. –Build programs that adhere to the protocol.
Session Management A290/A590, Fall /25/2014.
NFS. The Sun Network File System (NFS) An implementation and a specification of a software system for accessing remote files across LANs. The implementation.
.NET Mobile Application Development Remote Procedure Call.
7/15/2015B.RamamurthyPage 1 File System B. Ramamurthy.
CS140 Review Session Project 4 – File Systems Varun Arora Based on Vincenzo Di Nicola’s slide 7/16/2015cs140 Review Session1.
NETWORK FILE SYSTEM (NFS) By Ameeta.Jakate. NFS NFS was introduced in 1985 as a means of providing transparent access to remote file systems. NFS Architecture.
File Systems (2). Readings r Silbershatz et al: 11.8.
Network File System (NFS) Brad Karp UCL Computer Science CS GZ03 / M030 6 th, 7 th October, 2008.
Presented by: Alvaro Llanos E.  Motivation and Overview  Frangipani Architecture overview  Similar DFS  PETAL: Distributed virtual disks ◦ Overview.
Transactions and Reliability. File system components Disk management Naming Reliability  What are the reliability issues in file systems? Security.
Networked File System CS Introduction to Operating Systems.
SIMULATED UNIX FILE SYSTEM Implementation in C Tarek Youssef Bipanjit Sihra.
CS 390- Unix Programming Environment CS 390 Unix Programming Environment Topics to be covered: Distributed Computing Fundamentals.
FTP Client Application CSC 8560 Brian Jorgage 4/27/2004.
Object-Oriented Analysis & Design Subversion. Contents  Configuration management  The repository  Versioning  Tags  Branches  Subversion 2.
Spring/2002 Distributed Software Engineering C:\unocourses\4350\slides\DefiningThreads 1 RMI.
RMI remote method invocation. Traditional network programming The client program sends data to the server in some intermediary format and the server has.
CSE 451: Operating Systems Winter 2015 Module 22 Remote Procedure Call (RPC) Mark Zbikowski Allen Center 476 © 2013 Gribble, Lazowska,
Page 1 Remote Procedure Calls Paul Krzyzanowski Distributed Systems Except as otherwise noted, the content of this presentation.
1 Developing Application in Distributed Computing Environment (DCE)
Li Tak Sing COMPS311F. RMI callbacks In previous example, only the client can initiate a communication with the server. The server can only response to.
 Communication Distributed Systems IT332. Outline  Fundamentals  Layered network communication protocols  Types of communication  Remote Procedure.
Project 6 Unix File System. Administrative No Design Review – A design document instead 2-3 pages max No collaboration with peers – Piazza is for clarifications.
Jigsaw Performance Analysis Potential Bottlenecks.
Linux+ Guide to Linux Certification, Third Edition
Linux+ Guide to Linux Certification, Third Edition
GLOBAL EDGE SOFTWERE LTD1 R EMOTE F ILE S HARING - Ardhanareesh Aradhyamath.
Remote Procedure Call RPC
Lecture 24 Sun’s Network File System. PA3 In clkinit.c.
Lecture 4 Mechanisms & Kernel for NOSs. Mechanisms for Network Operating Systems  Network operating systems provide three basic mechanisms that support.
Manish Kumar,MSRITSoftware Architecture1 Remote procedure call Client/server architecture.
Computer Science Lecture 3, page 1 CS677: Distributed OS Last Class: Communication in Distributed Systems Structured or unstructured? Addressing? Blocking/non-blocking?
COT 4600 Operating Systems Fall 2009 Dan C. Marinescu Office: HEC 439 B Office hours: Tu-Th 3:00-4:00 PM.
Review CS File Systems - Partitions What is a hard disk partition?
RPC Model, Stubs and Skeletons Divya Nampalli
Open Science Grid Build a Grid Session Siddhartha E.S University of Florida.
CS140 Project 4 Due Thursday March 10th Slides adapted from Samir Selman’s Kiyoshi Shikuma.
Implementing Remote Procedure Call Landon Cox February 12, 2016.
Mutual Exclusion Algorithms. Topics r Defining mutual exclusion r A centralized approach r A distributed approach r An approach assuming an organization.
Chapter Five Distributed file systems. 2 Contents Distributed file system design Distributed file system implementation Trends in distributed file systems.
Computer Science Lecture 4, page 1 CS677: Distributed OS Last Class: RPCs RPCs make distributed computations look like local computations Issues: –Parameter.
Bigtable A Distributed Storage System for Structured Data.
Distributed Web Systems Distributed Objects and Remote Method Invocation Lecturer Department University.
4.3 Network File System (NFS)
GARRETT SINGLETARY.
File service architecture
File System B. Ramamurthy B.Ramamurthy 11/27/2018.
Issues in Client/Server Programming
Distribution Infrastructures
First slide Rest of project 2 due next Friday Today:
Chapter 15: File System Internals
Today: Distributed File Systems
Presentation transcript:

Intro to DFS assignment

Announcements DFS (PA4) Part 1 will be out over the weekend Part 1 Due 10/30 11:59 pm Part 2 12/7 11:59 pm Part 3 12/14 11:59 pm Part 4 (optional extra credit) 12/23 11:59pm

You build a DFS! 1. Lock server, at-most-once RPC semantics 2. Implement extent server; create/lookup/readdir FUSE ops 3. Implement read/write/open/setattr 4. Implement mkdir/unlink, integrate locks! (optional)

Programming Assignment: YFS Yet-another File System. :) Simpler version of what we just talked about: only one extent server (you don’t have to implement Petal; single lock server)

Part 1: lock server Implementing the lock server Provide mutual exclusion to ‘lock resources’ Using pthread mutexes, condition variables! Implementing at-most-once RPC semantics

Providing mutual exclusion over the net In the assignment: lock is a 64-bit number Client might: acquire(lock_a); do work; release(lock_a); You must create the lock if it doesn’t yet exist on the server

Example tester %./lock_tester 3772 simple lock client acquire a release a acquire a release a acquire a acquire b release b release a test2: client 0 acquire a release a test2: client 2 acquire a release a...

What’s provided Simple RPC framework, skeleton for lockserver code What it does: handles setting up sockets, marshalling/unmarshalling basic data types, sending over TCP (whew) What it doesn’t: does not keep track of RPC request state; duplicate RPCs are invoked twice! :(

Example RPC call

Your job for lock server

Lossy RPC_LOSSY: drops, duplicates, delays. Run lock_tester with RPC_LOSSY=0 Now try running it with RPC_LOSSY=5 Simulates network failure it should hang or error. FAIL!

Why? No at-most-once RPC semantics implemented...yet. If reply dropped, duplicate sent: acquire(a), acquire(a)

Strawman Remember every RPC call Client sends a unique RPC identifier Remember every RPC reply To avoid invoking the actual function What’s the problem with this approach?

Sliding window RPCs Assume the following sent by client: marshall m1 << clt nonce // client id << svr nonce // server id << proc // procedure number (bind, acquire, etc). << myxid // unique request id for this RPC << xid_rep_window.front() // Last out-of-order RPC reply received << req.str(); // Data But we need some additional info from the client

At the end % export RPC_LOSSY=5 %./lock_server 3772 & %./lock_tester all tests passed %./rpctest simple_tests... all tests passed

Helper functions Method_thread

Code reading

Part 2 Extent server FUSE! Semantics of filesystem calls

Programming Assignment: YFS Part 1: lock server Part 2: extent server

The extent server Don’t have to deal with storing data on disk: you will store contents in memory map All yfs_client operations synchronize with same extent server

Getattr (given)

Your job Extend extent_server to support put(extentid, string,...) get(extentid, string&) remove(extentid,...) See extent_protocol.h and extent_smain.cc Later you will likely add more! Must properly deal with ctime/atime/mtime

Data formats at extent server Directories maintain a mapping of filenames to inode numbers if root (inode 1) has two files “file1”, “file2”: get(1) might return file1: file2: You can choose how you store this information as a string

Metadata time mgmt atime: access time Updated whenever file contents accessed Set to 0 on file creation mtime: modification time Updated when file contents modified ctime: change time Updated whenever metadata modified

FUSE

Mapping FUSE functions

Example Fuse.cc void fuseserver_getattr(

CREATE/MKNOD Generate inode number (rand()) This is called on files: make sure inode has MSB set to 1 Directories have MSB set to 0 Create the file at the extent server Add the name to inode mapping to parent info at extent server

Lookup Given: filename, parent inode Look through parent directory list Find inode that maps from filename Getattr on that inode Fill in return structure (fuse_entry_param)

Readdir Given: inode number for a directory Get filename:inode mapping for that dir For each filename:inode pair call dirbuf_add(...) function provided

You will be able to create empty files in the root directory and do an ls on the root You’re not yet storing files...

Testing./start.sh./test-lab-2.pl./stop.sh test-lab-2.pl creates a bunch of files, tests whether the files the script created are there

Parts 3 and 4 Part 3: You implement open, read, write, setattr Part 4: You implement mkdir, unlink, and locking