Implementation of Simple Cloud-based Distributed File System Group ID: 4 Baolin Wu, Liushan Yang, Pengyu Ji.

Slides:



Advertisements
Similar presentations
Mapreduce and Hadoop Introduce Mapreduce and Hadoop
Advertisements

Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung
Simple Regenerating Codes: Network Coding for Cloud Storage Dimitris S. Papailiopoulos, Jianqiang Luo, Alexandros G. Dimakis, Cheng Huang, and Jin Li University.
Abstract HyFS: A Highly Available Distributed File System Jianqiang Luo, Mochan Shrestha, Lihao Xu Department of Computer Science, Wayne State University.
Distributed Systems 1 Topics  What is a Distributed System?  Why Distributed Systems?  Examples of Distributed Systems  Distributed System Requirements.
U NIVERSITY OF M ASSACHUSETTS, A MHERST Department of Computer Science Emery Berger University of Massachusetts Amherst Operating Systems CMPSCI 377 Lecture.
Team Wolf Distributed, Consistent and Secure USB Storage Final Project Review Eddie Lai Matt Dube Sean Busch Zhou Zheng.
1/19 Presented by: Maedeh Tashakkorian Supervisor: Hadi Salimi Mazandaran University of Science and Technology February, 2011.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved DISTRIBUTED SYSTEMS.
Hands-On Microsoft Windows Server 2003 Administration Chapter 5 Administering File Resources.
DISTRIBUTED COMPUTING
MetaSync File Synchronization Across Multiple Untrusted Storage Services Seungyeop Han Haichen Shen, Taesoo Kim*, Arvind Krishnamurthy,
Addition to Networking.  There is no unique and standard definition out there  Cloud Computing is a general term used to describe a new class of network.
Applied Architectures Eunyoung Hwang. Objectives How principles have been used to solve challenging problems How architecture can be used to explain and.
 Distributed Software Chapter 18 - Distributed Software1.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google∗
Distributed File Systems Concepts & Overview. Goals and Criteria Goal: present to a user a coherent, efficient, and manageable system for long-term data.
CSE 486/586, Spring 2012 CSE 486/586 Distributed Systems Distributed File Systems Steve Ko Computer Sciences and Engineering University at Buffalo.
Team Members Lora zalmover Roni Brodsky Academic Advisor Professional Advisors Dr. Natalya Vanetik Prof. Shlomi Dolev Dr. Guy Tel-Zur.
Distributed Storage System Survey
By: Jeffrey Dean & Sanjay Ghemawat Presented by: Warunika Ranaweera Supervised by: Dr. Nalin Ranasinghe.
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
Distributed Systems Principles and Paradigms Chapter 10 Distributed File Systems 01 Introduction 02 Communication 03 Processes 04 Naming 05 Synchronization.
Technology Overview. Agenda What’s New and Better in Windows Server 2003? Why Upgrade to Windows Server 2003 ?  From Windows NT 4.0  From Windows 2000.
9/14/2015B.Ramamurthy1 Operating Systems : Overview Bina Ramamurthy CSE421/521.
An Architecture for Video Surveillance Service based on P2P and Cloud Computing Yu-Sheng Wu, Yue-Shan Chang, Tong-Ying Juang, Jing-Shyang Yen speaker:
Map Reduce for data-intensive computing (Some of the content is adapted from the original authors’ talk at OSDI 04)
B.Ramamurthy9/19/20151 Operating Systems u Bina Ramamurthy CS421.
M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.
Unified Cloud Storage Group 9 Navneet Joshi Apoorva Gupta Gurinder Pal Singh.
TRƯỜNG ĐẠI HỌC CÔNG NGHỆ Bộ môn Mạng và Truyền Thông Máy Tính.
MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat.
Apache Cassandra - Distributed Database Management System Presented by Jayesh Kawli.
Distributed File System By Manshu Zhang. Outline Basic Concepts Current project Hadoop Distributed File System Future work Reference.
Large Scale Machine Translation Architectures Qin Gao.
Presenters: Rezan Amiri Sahar Delroshan
Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture The Metadata Consistency Model File Mutation.
INTRODUCTION TO DBS Database: a collection of data describing the activities of one or more related organizations DBMS: software designed to assist in.
11 CLUSTERING AND AVAILABILITY Chapter 11. Chapter 11: CLUSTERING AND AVAILABILITY2 OVERVIEW  Describe the clustering capabilities of Microsoft Windows.
Unified Cloud Storage Navneet Joshi, Apoorva Gupta, Gurinder Pal Singh Today there are a number of cloud storage services (Dropbox, Google drive, Box).
HADOOP DISTRIBUTED FILE SYSTEM HDFS Reliability Based on “The Hadoop Distributed File System” K. Shvachko et al., MSST 2010 Michael Tsitrin 26/05/13.
CS525: Big Data Analytics MapReduce Computing Paradigm & Apache Hadoop Open Source Fall 2013 Elke A. Rundensteiner 1.
Toward Efficient and Simplified Distributed Data Intensive Computing IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 22, NO. 6, JUNE 2011PPT.
Copyright © 2012 Cleversafe, Inc. All rights reserved. 1 Combining the Power of Hadoop with Object-Based Dispersed Storage.
The Sky.NET Framework COMP 410 April 22, Overview Brief overview of the current status of the Sky.Net FrameworkBrief overview of the current status.
Ch 11 Distributed File System Ch11.1 Architecture Lei Zhang Oct
Dsitributed File Systems
Implementation of Simple Cloud-based Distributed File System Group ID: 4 Baolin Wu, Liushan Yang, Pengyu Ji.
Distributed File System. Outline Basic Concepts Current project Hadoop Distributed File System Future work Reference.
Chapter Five Distributed file systems. 2 Contents Distributed file system design Distributed file system implementation Trends in distributed file systems.
Distributed Systems: Distributed File Systems Ghada Ahmed, PhD. Assistant Prof., Computer Science Dept. Web:
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Presenter: Chao-Han Tsai (Some slides adapted from the Google’s series lectures)
Seminar On Rain Technology
Cloud-based movie search web application with transaction service Group 14 Yuanfan Zhang Ji Zhang Zhuomeng Li.
COMP7330/7336 Advanced Parallel and Distributed Computing MapReduce - Introduction Dr. Xiao Qin Auburn University
Pouya Ostovari and Jie Wu Computer & Information Sciences
Second-hand Trading Web Service Group Member: Jingwei Hao Xiaofeng Yuan Yanjun Liu.
Advanced Operating Systems Chapter 6.1 – Characteristics of a DFS Jongchan Shin.
Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung
Introduction to Distributed Platforms
CSC-8320 Advanced Operating System
Replication Middleware for Cloud Based Storage Service
Sajitha Naduvil-vadukootu
DUCKS – Distributed User-mode Chirp-Knowledgeable Server
Ch 11 Distributed File System
Operating Systems Bina Ramamurthy CSE421 11/27/2018 B.Ramamurthy.
Outline Midterm results summary Distributed file systems – continued
Operating Systems : Overview
DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S
Distributed Systems (15-440)
Presentation transcript:

Implementation of Simple Cloud-based Distributed File System Group ID: 4 Baolin Wu, Liushan Yang, Pengyu Ji

Motivation & Goals Distributed File System provides more reliable service. Cloud Storage Service is very popular. Google Cloud Storage, Dropbox, Amazon SC3 have many users. Many users have more than one cloud storage service accounts. But there is no unified way of managing all the accounts. Currently some products provide the solution that treats different accounts as different folder but a single file can only be stored in one of the platform. Our project is to combine different cloud platforms together and give users the file system’s perspective about the files stored in cloud. Such distributed platform has following advantages: 1. Easier to manage the files. 2. Distributed File System will give user more robust file system. Single node cloud service failure will not bring the whole system down because files are sliced, replicated and stored in different cloud products. 3. More space is available by combining different products together. 4. Can enhance security of the stored file. Different file segments can be encrypted using different keys. It is harder to know the file contents even under compromised server. (add-on feature if we have enough time) Goals: Fault-tolerant file system transparent to user. Better space utilization of different cloud storage service and easy user interface.

Related Work and Reference Some important papers: Ghemawat, Sanjay, Howard Gobioff, and Shun-Tak Leung. "The Google file system." ACM SIGOPS operating systems review. Vol. 37. No. 5. ACM, Shvachko, Konstantin, et al. "The Hadoop distributed file system." Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on. IEEE, Cloud Storage Product: The cloud service includes Google Storage, DropBox, Box, Amazon SC3 etc. The APIs used to manipulate the files in the Storage Products: Google Storage: DropBox: Some products give user File System’s view of single cloud storage service: FUSE file system for Google Drive: Dropbox file system for Linux: Some products view different cloud service as separate folder, most of them are commercial: Otixo: Jolicloud: Mover:

System Architecture 1.To simplify the communication between client and server, we use HTTP protocol. 2.Master server only care about transaction and meta information. No upload/download happens to server. Single Master node with mechanism to recover like what Google File System does. 3.Client sends request to master server for reading/writing. Master approves/declines the request. Receiving the approve token, the client calls the cloud service directly.

Implementation & Evaluation Plan WeekMilestone 3Draft of User Mode File System, Cloud APIs, MetaInfo Requestor 4User File Processor, MetaInfo Requestor and MetaInfo Storage. 5File replication algorithm, Single User file synchronization 6Fault Tolerance Evaluation with single user. 7Support Concurrent Modification of file and Transaction Manager. 8Fault-tolerance with single Master server. 9-10Potential Improvement (Encryption etc.), Project Report. 1.We first implement the client by calling the server function directly. Then separate server and client and bring the HTTP communication into the system. 2.Before implementing the File system, we will use many test cases simulating file create/read/write, then mount the file system and manipulate files in the system. 3.Try to evaluate some failure case where client, server or cloud platform crashes. But there is no way to evaluate every possible failure cases due to the time and complexity.