NTU Cloud 2010/05/30. System Diagram Architecture Gluster File System – Provide a distributed shared file system for migration NFS – A Prototype Image.

Slides:

Advertisements

Similar presentations

MapReduce Online Created by: Rajesh Gadipuuri Modified by: Ying Lu.

Advertisements

Locality-Aware Dynamic VM Reconfiguration on MapReduce Clouds Jongse Park, Daewoo Lee, Bokyeong Kim, Jaehyuk Huh, Seungryoul Maeng.

Xen , Linux Vserver , Planet Lab

An Approach to Secure Cloud Computing Architectures By Y. Serge Joseph FAU security Group February 24th, 2011.

Look Who’s Talking: Discovering Dependencies between Virtual Machines Using CPU Utilization HotCloud 10 Presented by Xin.

Chapter 1: Introduction

Many kinds of clients and servers This work is licensed under a Creative Commons Attribution-Noncommercial- Share Alike 3.0 License. Skills: none IT concepts:

COMS E Cloud Computing and Data Center Networking Sambit Sahu

DESIGN CONSIDERATIONS OF A GEOGRAPHICALLY DISTRIBUTED IAAS CLOUD ARCHITECTURE CS 595 LECTURE 10 3/20/2015.

1/16/2008CSCI 315 Operating Systems Design1 Introduction Notice: The slides for this lecture have been largely based on those accompanying the textbook.

Virtualization Infrastructure Administration Cluster Jakub Yaghob.

VIRTUALISATION OF HADOOP CLUSTERS Dr G Sudha Sadasivam Assistant Professor Department of CSE PSGCT.

Presented by : Ran Koretzki. Basic Introduction What are VM’s ? What is migration ? What is Live migration ?

Cyberaide Virtual Appliance: On-demand Deploying Middleware for Cyberinfrastructure Tobias Kurze, Lizhe Wang, Gregor von Laszewski, Jie Tao, Marcel Kunze,

Cloud Computing for the Enterprise November 18th, This work is licensed under a Creative Commons.

 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.

PRAGMA20 – PRAGMA 21 Collaborative Activities Resources Working Group.

Nimbus & OpenNebula Young Suk Moon. Nimbus - Intro Open source toolkit Provides virtual workspace service (Infrastructure as a Service) A client uses.

Virtualization. Virtualization  In computing, virtualization is a broad term that refers to the abstraction of computer resources  It is "a technique.

Projects. High Performance Computing Projects Design and implement an HPC cluster with one master node and two compute nodes. (Hint: use Rocks HPC Cluster.

+ CS 325: CS Hardware and Software Organization and Architecture Cloud Architectures.

UI and Data Entry UI and Data Entry Front-End Business Logic Mid-Tier Data Store Back-End.

INTRODUCTION TO CLOUD COMPUTING CS 595 LECTURE 2.

Presented by: Sanketh Beerabbi University of Central Florida COP Cloud Computing.

A study of introduction of the virtualization technology into operator consoles T.Ohata, M.Ishii / SPring-8 ICALEPCS 2005, October 10-14, 2005 Geneva,

How computer’s are linked together.

Large Scale Sky Computing Applications with Nimbus Pierre Riteau Université de Rennes 1, IRISA INRIA Rennes – Bretagne Atlantique Rennes, France

COMS E Cloud Computing and Data Center Networking Sambit Sahu

N. GSU Slide 1 Chapter 05 Clustered Systems for Massive Parallelism N. Xiong Georgia State University.

Luis Russi¹, Carlos R. Senna¹, Edmundo R. M. Madeira¹, Xuan Liu², Shuai Zhao², and Deep Medhi² Hadoop-in-a-Hybrid-Cloud GEC21 The 21st GENI Engineering.

Synchronized Co-migration of Virtual Machines for IDS Offloading in Clouds Kenichi Kourai and Hisato Utsunomiya Kyushu Institute of Technology, Japan.

20409A 7: Installing and Configuring System Center 2012 R2 Virtual Machine Manager Module 7 Installing and Configuring System Center 2012 R2 Virtual.

Virtual cloud R 陳昌毅 R 顏昭恩 R 黃伯淳 2010/06/03.

Architecture & Cybersecurity – Module 3 ELO-100Identify the features of virtualization. (Figure 3) ELO-060Identify the different components of a cloud.

OpenNebula: Experience at SZTAKI Peter Kacsuk, Sandor Acs, Mark Gergely, Jozsef Kovacs MTA SZTAKI EGI CF Helsinki.

Next Generation of Apache Hadoop MapReduce Owen

PARALLEL AND DISTRIBUTED PROGRAMMING MODELS U. Jhashuva 1 Asst. Prof Dept. of CSE om.

Auxiliary services Web page Secrets repository RSV Nagios Monitoring Ganglia NIS server Syslog Forward FermiCloud: A private cloud to support Fermilab.

IMPROVEMENT OF COMPUTATIONAL ABILITIES IN COMPUTING ENVIRONMENTS WITH VIRTUALIZATION TECHNOLOGIES Abstract We illustrates the ways to improve abilities.

Information Initiative Center, Hokkaido University North 11, West 5, Sapporo , Japan Tel, Fax: General.

INFN/IGI contributions Federated Clouds Task Force F2F meeting November 24, 2011, Amsterdam.

WP5 – Infrastructure Operations Test and Production Infrastructures StratusLab kick-off meeting June 2010, Orsay, France GRNET.

VIRTUAL MACHINE – VMWARE. VIRTUAL MACHINE (VM) What is a VM? – A virtual machine (VM) is a software implementation of a computing environment in which.

Chapter 1: Introduction What is an Operating System? Mainframe Systems Desktop Systems Multiprocessor Systems Distributed Systems Clustered System Real.

CLOUD ARCHITECTURE Many organizations and researchers have defined the architecture for cloud computing. Basically the whole system can be divided into.

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING CLOUD COMPUTING

Chapter 1: Introduction

Chapter 1: Introduction

The advances in IHEP Cloud facility

NEWS LAB 薛智文嵌入式系統暨無線網路實驗室

Blueprint of Persistent Infrastructure as a Service

Dag Toppe Larsen UiB/CERN CERN,

Dag Toppe Larsen UiB/CERN CERN,

StratusLab Final Periodic Review

StratusLab Final Periodic Review

Chapter 1: Introduction

Chapter 1: Introduction

Chapter 1: Introduction

Chapter 1: Introduction

Chapter 1: Introduction

20409A 7: Installing and Configuring System Center 2012 R2 Virtual Machine Manager Module 7 Installing and Configuring System Center 2012 R2 Virtual.

Introduction to Apache

Chapter 1: Introduction

CS 345A Data Mining MapReduce This presentation has been altered.

Chapter 1: Introduction

Chapter 1: Introduction

Chapter 1: Introduction

Chapter 1: Introduction

Client/Server Computing and Web Technologies

Chapter 1: Introduction

Presentation transcript:

NTU Cloud 2010/05/30

System Diagram

Architecture Gluster File System – Provide a distributed shared file system for migration NFS – A Prototype Image storage space Node Gluster File System Compute ImgC- Img S- ImgStorage ImgC- Img NFS Prototype Img

Architecture Prototype Image – Original Image e.g. Hadoop MPI Compute Image – Modified Images for user – Do not preserve the content after cluster shutdown Node Gluster File System Compute ImgC- Img S- ImgStorage ImgC- Img NFS Prototype Img

XEN A hypervisor Virtualization

Cloud Master Monitor system state Scheduling Use NFS to store Prototype Image Web server

OpenNebula A middleware Provides an interface to manage virtual infrastructure (computation and network) VM Migration => We use OpenNebula to manage VM deployment, migration and set up virtual local area network(VLAN).

Gluster file system User level distributed file system Client/Server Architecture Use TCP/IP to transfer data =>We use GlusterFS to build our share file system environment for VM live migration. =>Our deployment is "symmetrical" - every machine is both a server and a client.

System Flow

Hadoop Benchmark Case 1 – M1 : Master + Slave-01 + Slave02 Case 2 – M1 : Master – M2 : Slave-01 + Slave-02 Case 3 – M1 : Master – M3 : Slave-01 + Slave-02 Case 4 – M1 : Master – M2 : Slave-01 – M3 : Slave-02

All in M1Slave in M2Slave in M3Slave-01 in M2 Slave-02 in M Sec Sec Iteration

Set 1 VMHost MachineVCPUMemPurpose Set 1.1Single machine MasterM112.2GNamenode+Datanode+Jobtracker+Tasktracker WorkerM111.2GDatanode+Tasktracker Set 1.2Two machine MasterM112.2GNamenode+Datanode+Jobtracker+Tasktracker WorkerM211.2GDatanode+Tasktracker M1&M2 has same CPU and Memory size. HADOOP_HEAPSIZE=500MB mapred.child.java.opts=100MB RandomWriter 10M for 30Maps Sortting HDFS_BYTES_READ= HDFS_BYTES_WRITTEN=

Sort Therefore, putting two VM into one machine performance slowdown to 88.92% two machine / single machine = % Launched reduce tasks=4 Others=3 Reduce shuffle bytes= Reduce shuffle bytes= Exactly the same!

Set 2 VMHost MachineVCPUMemPurpose Set 2.1Single machine MasterM122.2GNamenode+Datanode+Jobtracker+Tasktracker WorkerM121.2GDatanode+Tasktracker Set 2.2Two machine MasterM122.2GNamenode+Datanode+Jobtracker+Tasktracker WorkerM221.2GDatanode+Tasktracker 1.RandomWriter10M for 30Maps 2.Sort HADOOP_HEAPSIZE=500MB mapred.child.java.opts=100MB

RandomWriter Therefore, putting two VM into one machine performance slowdown to 80.70% two machine / single machine = %

RandomWriter Single machineTwo machine IterationSecHDFS_BYTES_WRITTENSecHDFS_BYTES_WRITTEN Avg Avg. on 1,2,

Sort

Current Progress Xen 4.0 is ready on each node. We can offer two kinds of images – Hadoop – MPI Start up VMs to destination node automatically. Configure MPI and Hadoop environment for use automatically.