EFFECTIVE LOAD-BALANCING VIA MIGRATION AND REPLICATION IN SPATIAL GRIDS ANIRBAN MONDAL KAZUO GODA MASARU KITSUREGAWA INSTITUTE OF INDUSTRIAL SCIENCE UNIVERSITY.

Slides:



Advertisements
Similar presentations
Investigating Distributed Caching Mechanisms for Hadoop Gurmeet Singh Puneet Chandra Rashid Tahir.
Advertisements

Distributed Systems Major Design Issues Presented by: Christopher Hector CS8320 – Advanced Operating Systems Spring 2007 – Section 2.6 Presentation Dr.
P2PR-tree: An R-tree-based Spatial Index for P2P Environments ANIRBAN MONDAL YI LIFU MASARU KITSUREGAWA University of Tokyo.
LIBRA: Lightweight Data Skew Mitigation in MapReduce
High Performance Computing Course Notes Grid Computing.
Serverless Network File Systems. Network File Systems Allow sharing among independent file systems in a transparent manner Mounting a remote directory.
Study of Hurricane and Tornado Operating Systems By Shubhanan Bakre.
Tradeoffs in Scalable Data Routing for Deduplication Clusters FAST '11 Wei Dong From Princeton University Fred Douglis, Kai Li, Hugo Patterson, Sazzala.
1 Routing and Scheduling in Web Server Clusters. 2 Reference The State of the Art in Locally Distributed Web-server Systems Valeria Cardellini, Emiliano.
Atomistic Protein Folding Simulations on the Submillisecond Timescale Using Worldwide Distributed Computing Qing Lu CMSC 838 Presentation.
Locality-Aware Request Distribution in Cluster-based Network Servers 1. Introduction and Motivation --- Why have this idea? 2. Strategies --- How to implement?
Chapter 1 Introduction 1.1A Brief Overview - Parallel Databases and Grid Databases 1.2Parallel Query Processing: Motivations 1.3Parallel Query Processing:
1 Introduction to Load Balancing: l Definition of Distributed systems. Collection of independent loosely coupled computing resources. l Load Balancing.
Introspective Replica Management Yan Chen, Hakim Weatherspoon, and Dennis Geels Our project developed and evaluated a replica management algorithm suitable.
1 By Vanessa Newey. 2 Introduction Background Scalability in Distributed Simulation Traditional Aggregation Techniques Problems with Traditional Methods.
12006/9/26 Load Balancing in Dynamic Structured P2P Systems Brighten Godfrey, Karthik Lakshminarayanan, Sonesh Surana, Richard Karp, Ion Stoica INFOCOM.
On Fairness, Optimizing Replica Selection in Data Grids Husni Hamad E. AL-Mistarihi and Chan Huah Yong IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,
Dynamic Load Sharing and Balancing Sig Freund. Outline Introduction Distributed vs. Traditional scheduling Process Interaction models Distributed Systems.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Moving Objects Databases Nilanshu Dharma Shalva Singh.
Design and Implementation of a Single System Image Operating System for High Performance Computing on Clusters Christine MORIN PARIS project-team, IRISA/INRIA.
Roger ZimmermannCOMPSAC 2004, September 30 Spatial Data Query Support in Peer-to-Peer Systems Roger Zimmermann, Wei-Shinn Ku, and Haojun Wang Computer.
Abstract Load balancing in the cloud computing environment has an important impact on the performance. Good load balancing makes cloud computing more.
1 Distributed Operating Systems and Process Scheduling Brett O’Neill CSE 8343 – Group A6.
KNR-tree: A novel R-tree-based index for facilitating Spatial Window Queries on any k relations among N spatial relations in Mobile environments ANIRBAN.
Presenter: Dipesh Gautam.  Introduction  Why Data Grid?  High Level View  Design Considerations  Data Grid Services  Topology  Grids and Cloud.
Grid Data Management A network of computers forming prototype grids currently operate across Britain and the rest of the world, working on the data challenges.
A Lightweight Platform for Integration of Resource Limited Devices into Pervasive Grids Stavros Isaiadis and Vladimir Getov University of Westminster
Peer to Peer Research survey TingYang Chang. Intro. Of P2P Computers of the system was known as peers which sharing data files with each other. Build.
Cloud Computing Energy efficient cloud computing Keke Chen.
The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Dataset Caitlin Minteer & Kelly Clynes.
Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.
Scalable Web Server on Heterogeneous Cluster CHEN Ge.
Peer-to-Peer Distributed Shared Memory? Gabriel Antoniu, Luc Bougé, Mathieu Jan IRISA / INRIA & ENS Cachan/Bretagne France Dagstuhl seminar, October 2003.
Your university or experiment logo here Caitriana Nicholson University of Glasgow Dynamic Data Replication in LCG 2008.
1 Distributed Energy-Efficient Scheduling for Data-Intensive Applications with Deadline Constraints on Data Grids Cong Liu and Xiao Qin Auburn University.
The Owner Share scheduler for a distributed system 2009 International Conference on Parallel Processing Workshops Reporter: 李長霖.
Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.
Investigating Survivability Strategies for Ultra-Large Scale (ULS) Systems Vanderbilt University Nashville, Tennessee Institute for Software Integrated.
1 ACTIVE FAULT TOLERANT SYSTEM for OPEN DISTRIBUTED COMPUTING (Autonomic and Trusted Computing 2006) Giray Kömürcü.
Computer Science and Engineering Predicting Performance for Grid-Based P. 1 IPDPS’07 A Performance Prediction Framework.
Data Replication and Power Consumption in Data Grids Susan V. Vrbsky, Ming Lei, Karl Smith and Jeff Byrd Department of Computer Science The University.
Caitriana Nicholson, CHEP 2006, Mumbai Caitriana Nicholson University of Glasgow Grid Data Management: Simulations of LCG 2008.
A Fault-Tolerant Environment for Large-Scale Query Processing Mehmet Can Kurt Gagan Agrawal Department of Computer Science and Engineering The Ohio State.
1 THE EARTH SIMULATOR SYSTEM By: Shinichi HABATA, Mitsuo YOKOKAWA, Shigemune KITAWAKI Presented by: Anisha Thonour.
Distributed System Services Fall 2008 Siva Josyula
A Grid-enabled Multi-server Network Game Architecture Tianqi Wang, Cho-Li Wang, Francis C.M.Lau Department of Computer Science and Information Systems.
DISTRIBUTED COMPUTING
Distributed Computing Systems CSCI 4780/6780. Scalability ConceptExample Centralized servicesA single server for all users Centralized dataA single on-line.
Dynamic Scheduling Monte-Carlo Framework for Multi-Accelerator Heterogeneous Clusters Authors: Anson H.T. Tse, David B. Thomas, K.H. Tsoi, Wayne Luk Source:
Ohio State University Department of Computer Science and Engineering Servicing Range Queries on Multidimensional Datasets with Partial Replicas Li Weng,
Data Consolidation: A Task Scheduling and Data Migration Technique for Grid Networks Author: P. Kokkinos, K. Christodoulopoulos, A. Kretsis, and E. Varvarigos.
GPFS: A Shared-Disk File System for Large Computing Clusters Frank Schmuck & Roger Haskin IBM Almaden Research Center.
On Improving the Performance Dependability of Unstructured P2P Systems via Replication ANIRBAN MONDAL YI LIFU MASARU KITSUREGAWA Institute of Industrial.
University of Texas at Arlington Scheduling and Load Balancing on the NASA Information Power Grid Sajal K. Das, Shailendra Kumar, Manish Arora Department.
Load Rebalancing for Distributed File Systems in Clouds.
Cluster computing. 1.What is cluster computing? 2.Need of cluster computing. 3.Architecture 4.Applications of cluster computing 5.Advantages of cluster.
COMP7500 Advanced Operating Systems I/O-Aware Load Balancing Techniques Dr. Xiao Qin Auburn University
System Components Operating System Services System Calls.
Grid Services for Digital Archive Tao-Sheng Chen Academia Sinica Computing Centre
ScotGRID is the Scottish prototype Tier 2 Centre for LHCb and ATLAS computing resources. It uses a novel distributed architecture and cutting-edge technology,
Anirban Mondal (IIS, University of Tokyo, JAPAN)
Introduction to Load Balancing:
Distributed Network Traffic Feature Extraction for a Real-time IDS
Grid Computing.
Auburn University COMP7500 Advanced Operating Systems I/O-Aware Load Balancing Techniques (2) Dr. Xiao Qin Auburn University.
Li Weng, Umit Catalyurek, Tahsin Kurc, Gagan Agrawal, Joel Saltz
Cloud Computing Architecture
Specialized Cloud Architectures
L. Glimcher, R. Jin, G. Agrawal Presented by: Leo Glimcher
Presentation transcript:

EFFECTIVE LOAD-BALANCING VIA MIGRATION AND REPLICATION IN SPATIAL GRIDS ANIRBAN MONDAL KAZUO GODA MASARU KITSUREGAWA INSTITUTE OF INDUSTRIAL SCIENCE UNIVERSITY OF TOKYO, JAPAN

PRESENTATION OUTLINE INTRODUCTION INTRODUCTION RELATED WORK RELATED WORK SYSTEM OVERVIEW SYSTEM OVERVIEW MIGRATION AND REPLICATION MIGRATION AND REPLICATION LOAD-BALANCING LOAD-BALANCING PERFORMANCE STUDY PERFORMANCE STUDY CONCLUSION AND FUTURE WORK CONCLUSION AND FUTURE WORK

INTRODUCTION Prevalence of spatial applications  GIS, CAD,VLSI  Resource management, development planning, emergency planning, scientific research, GIS, CAD,VLSI Unprecedented growth of available spatial data at geographically distributed locations  the need for efficient networking Emergence of GRID computing and powerful networks Motivates the design of a SPATIAL GRID.

CHALLENGES Scale Scale Heterogeneity Heterogeneity Dynamism Dynamism Cross-domain administrative issues Cross-domain administrative issues Efficient search and load-balancing mechanisms Efficient search and load-balancing mechanisms  We focus on load-balancing.  Load-balancing in GRIDs is much more complicated than in traditional environments.

LOAD-BALANCING Some nodes become hot Some nodes become hot  Skewed Workloads  Dynamic access patterns These hot nodes become bottlenecks These hot nodes become bottlenecks  Increased waiting times  High response times MAIN CONTRIBUTIONS MAIN CONTRIBUTIONS   Viewing a spatial GRID as comprising several clusters   Each cluster is a LAN   Proposal of an inter-cluster load-balancing algorithm which uses migration/replication of data.   Presentation of a scalable technique for dynamic data placement.

RELATED WORK Ongoing GRID projects   Earth Systems Grid (ESG)   NASA Information Power Grid (IPG)   Grid Physics Network (GriPhyN)   European DataGrid. [Thain01] Binding of execution and storage sites together into I/O communities [Thain01] Data-movement system (Kangaroo) Load-balancing Load-balancing  STATIC (BUBBA, tile technique)  DYNAMIC (Disk cooling) Job (Process) MIGRATION in CONDOR Job (Process) MIGRATION in CONDOR Spatial indexes: R-tree [Guttman:84]

SYSTEM OVERVIEW Viewing the GRID as a set of clusters Viewing the GRID as a set of clusters Distance between two clusters Distance between two clusters  Communication time between cluster leaders Neighbours Neighbours Definition of Load Definition of Load  Number of disk I/Os in a certain time interval  Normalize w.r.t CPU power Cluster leaders Cluster leaders  Coordinate cluster activities  Maintain meta-information  Data stored at its own cluster & its neighbours Hotspot detection via access statistics Hotspot detection via access statistics  Use only recent statistics

DATA MOVEMENT IN GRIDs MIGRATION & REPLICATION MIGRATION & REPLICATION  Unlike replication, migration implies deletion of hot data at the source node. Which option is better: Migration or Replication Which option is better: Migration or Replication  Load-balancing  Data Availability  Disk space usage  Periodic cleanup REPLICA CONSISTENCY ?? REPLICA CONSISTENCY ?? Decisions concerning migration/replication should be taken during run-time. Decisions concerning migration/replication should be taken during run-time.

DATA MOVEMENT (Cont.) Impact of heterogeneity on data movement Impact of heterogeneity on data movement  Administrative policies (e.g., security)  Data management techniques (Indexing, hotspot detection, etc)  CPU  Disk space Moving data entails movement of indexes. Moving data entails movement of indexes. To address variations in indexing schemes, we extract data from the index at a node and rebuild the index at the destination node. To address variations in indexing schemes, we extract data from the index at a node and rebuild the index at the destination node. Each node has two indexes Each node has two indexes  Index for its own data  Index for moved data

DATA MOVEMENT (Cont.) Impact of variations in disk space on data movement   ‘Pushing’ non-hot data to large capacity peers   Large-sized data: migration   Small-sized data: replication   Replicating small-sized hot data at small capacity peers   Large-sized hot data: migration to large capacity peers if peers are available, otherwise replication. Deletion of infrequently accessed replicas

INTER-CLUSTER LOAD-BALANCING Periodic exchange of load info between neighbours Periodic exchange of load info between neighbours Leader L considers itself to be overloaded if its load exceeds that of its neighbours by 10%. Leader L considers itself to be overloaded if its load exceeds that of its neighbours by 10%.  L determines its hot regions and informs its neighbours about disk space requirement of hot regions.  Number of hot regions depends upon load imbalance. Neighbours with enough disk space reply to L with their load status and disk space information. Neighbours with enough disk space reply to L with their load status and disk space information. These leaders are sorted (asc) in List1 based on their loads. These leaders are sorted (asc) in List1 based on their loads. L assigns hot regions to members of List 1 in a round-robin manner. L assigns hot regions to members of List 1 in a round-robin manner.  The hottest region is moved to first member of List1, the second hottest region is moved to second member of List1 and so on.

PERFORMANCE STUDY 16 SUN workstations, each of which is a 143 MHz Sun UltraSparc I processor (256 MB RAM) running Solaris operating system. These are connected by relatively high speed switch (200 Mbyte/s), the APnet. Each cluster is modeled by a workstation node. We simulated a transfer rate of 1 Mbit/second among the clusters. We implemented an R-tree on each of the clusters to organize the data allocated to each cluster. A real dataset (Greece Roads) Each cluster had more than data rectangles. Zipf distribution was used to model workload skews. We investigated only migration in this proposal.

PERFORMANCE OF OUR PROPOSED SCHEME

SNAPSHOT OF LOAD-BALANCING FOR ZIPF FACTOR OF 0.1

VARIATIONS IN WORKLOAD SKEW

SNAPSHOT OF LOAD DISTRIBUTION FOR ZIPF FACTOR OF 0.5

Huge amounts of available spatial data worldwide coupled with the emergence of GRID technologies and powerful networks motivate the design of a spatial GRID. Huge amounts of available spatial data worldwide coupled with the emergence of GRID technologies and powerful networks motivate the design of a spatial GRID. For performance reasons, effective load-balancing is necessary in such a spatial GRID. For performance reasons, effective load-balancing is necessary in such a spatial GRID. We view a GRID as a set of clusters. We view a GRID as a set of clusters. Proposal of a dynamic inter-cluster load-balancing strategy via migration/replication in GRIDs Proposal of a dynamic inter-cluster load-balancing strategy via migration/replication in GRIDs SUMMARY

FUTURE SCOPE OF WORK FAIRNESS IN LOAD-BALANCING FAIRNESS IN LOAD-BALANCING GRANULARITY OF DATA MOVEMENT GRANULARITY OF DATA MOVEMENT DETAILED PERFORMANCE STUDY DETAILED PERFORMANCE STUDY  REPLICATION  DIFFERENT WORKLOAD TYPES  SCALABILITY  INTEGRATION INTO EXISTING GRIDs