Wide Area Events Using DDS We have a model that can efficiently support a family of applications, Publish-Subscribe-Notify. To realize this model, we implemented.

Slides:



Advertisements
Similar presentations
Database Architectures and the Web
Advertisements

Dr. Kalpakis CMSC 621, Advanced Operating Systems. Fall 2003 URL: Distributed System Architectures.
Sensor Network Platforms and Tools
Study of Hurricane and Tornado Operating Systems By Shubhanan Bakre.
Distributed Storage March 12, Distributed Storage What is Distributed Storage?  Simple answer: Storage that can be shared throughout a network.
An Event Model for Internet Services Subscribe/Unsubscribe Post e e e e CHANNEL e1 e2 e3 e FILTER RPC (Work) e Push Pull TRANSDUCER SINK e Interface: postIF:
1 Principles of Reliable Distributed Systems Tutorial 12: Frangipani Spring 2009 Alex Shraer.
Contiki A Lightweight and Flexible Operating System for Tiny Networked Sensors Presented by: Jeremy Schiff.
CS 623 Lecture #9 Yen-Yu Chen Utku Irmak. Papers to be read Better operating system features for faster network servers.Better operating system features.
Active Messages: a Mechanism for Integrated Communication and Computation von Eicken et. al. Brian Kazian CS258 Spring 2008.
Computer Science Lecture 2, page 1 CS677: Distributed OS Last Class: Introduction Distributed Systems – A collection of independent computers that appears.
Overview of Lustre ECE, U of MN Changjin Hong (Prof. Tewfik’s group) Monday, Aug. 19, 2002.
A. Frank - P. Weisberg Operating Systems Introduction to Tasks/Threads.
Database System Architectures  Client-server Database System  Parallel Database System  Distributed Database System Wei Jiang.
File Systems (2). Readings r Silbershatz et al: 11.8.
Kevin Kline, SQL Sentry Director of Engineering Services, Microsoft SQL Server MVP since 2003 Twitter, Facebook, KEKline.
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc
Distributed File Systems Concepts & Overview. Goals and Criteria Goal: present to a user a coherent, efficient, and manageable system for long-term data.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.
Larisa kocsis priya ragupathy
The Center for Autonomic Computing is supported by the National Science Foundation under Grant No NSF CAC Seminannual Meeting, October 5 & 6,
HBase A column-centered database 1. Overview An Apache project Influenced by Google’s BigTable Built on Hadoop ▫A distributed file system ▫Supports Map-Reduce.
By Lecturer / Aisha Dawood 1.  You can control the number of dispatcher processes in the instance. Unlike the number of shared servers, the number of.
Node.js - What is Node.js? -
1 Moshe Shadmon ScaleDB Scaling MySQL in the Cloud.
1 Yasin N. Silva Arizona State University This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Distributed File Systems Overview  A file system is an abstract data type – an abstraction of a storage device.  A distributed file system is available.
CEPH: A SCALABLE, HIGH-PERFORMANCE DISTRIBUTED FILE SYSTEM S. A. Weil, S. A. Brandt, E. L. Miller D. D. E. Long, C. Maltzahn U. C. Santa Cruz OSDI 2006.
Andrew S. Budarevsky Adaptive Application Data Management Overview.
Introduction to DFS. Distributed File Systems A file system whose clients, servers and storage devices are dispersed among the machines of a distributed.
Presenters: Rezan Amiri Sahar Delroshan
DISTRIBUTED COMPUTING Introduction Dr. Yingwu Zhu.
Copyright © cs-tutorial.com. Overview Introduction Architecture Implementation Evaluation.
The Replica Location Service The Globus Project™ And The DataGrid Project Copyright (c) 2002 University of Chicago and The University of Southern California.
Disco : Running commodity operating system on scalable multiprocessor Edouard et al. Presented by Vidhya Sivasankaran.
Robustness in the Salus scalable block store Yang Wang, Manos Kapritsos, Zuocheng Ren, Prince Mahajan, Jeevitha Kirubanandam, Lorenzo Alvisi, and Mike.
Intel Research & Development ETA: Experience with an IA processor as a Packet Processing Engine HP Labs Computer Systems Colloquium August 2003 Greg Regnier.
Distributed File Systems
Simics: A Full System Simulation Platform Synopsis by Jen Miller 19 March 2004.
Resilient Distributed Datasets: A Fault- Tolerant Abstraction for In-Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave,
Copyright © 2006, GemStone Systems Inc. All Rights Reserved. Increasing computation throughput with Grid Data Caching Jags Ramnarayan Chief Architect GemStone.
AMQP, Message Broker Babu Ram Dawadi. overview Why MOM architecture? Messaging broker like RabbitMQ in brief RabbitMQ AMQP – What is it ?
6.894: Distributed Operating System Engineering Lecturers: Frans Kaashoek Robert Morris
A N I N - MEMORY F RAMEWORK FOR E XTENDED M AP R EDUCE 2011 Third IEEE International Conference on Coud Computing Technology and Science.
CS426: Building Decentralized Systems Mahesh Balakrishnan.
Bigtable: A Distributed Storage System for Structured Data
Large Scale Sharing Marco F. Duarte COMP 520: Distributed Systems September 19, 2004.
Gaia An Infrastructure for Active Spaces Prof. Klara Nahrstedt Prof. David Kriegman Prof. Dennis Mickunas
State Machine Replication State Machine Replication through transparent distributed protocols State Machine Replication through a shared log.
Distributed File System. Outline Basic Concepts Current project Hadoop Distributed File System Future work Reference.
CS294, YelickDataStructs, p1 CS Distributed Data Structures
Software Architecture of Sensors. Hardware - Sensor Nodes Sensing: sensor --a transducer that converts a physical, chemical, or biological parameter into.
Last Class: Introduction
Open Source distributed document DB for an enterprise
Oracle 11g Real Application Clusters Advanced Administration
Noah Treuhaft UC Berkeley ROC Group ROC Retreat, January 2002
Database Architectures and the Web
Replication Middleware for Cloud Based Storage Service
The Single Node B-tree for Highly Concurrent Distributed Data Structures by Barbara Hohlt 11/23/2018.
Hadoop Technopoints.
CSE 451: Operating Systems Winter Module 22 Distributed File Systems
Distributed File Systems
Distributed File Systems
CSE 451: Operating Systems Spring Module 21 Distributed File Systems
Distributed File Systems
CSE 451: Operating Systems Winter Module 22 Distributed File Systems
Chapter 15: File System Internals
Distributed File Systems
Distributed File Systems
Presentation transcript:

Wide Area Events Using DDS We have a model that can efficiently support a family of applications, Publish-Subscribe-Notify. To realize this model, we implemented Distributed Data Structures [GBHC2000] with B-trees.We have a model that can efficiently support a family of applications, Publish-Subscribe-Notify. To realize this model, we implemented Distributed Data Structures [GBHC2000] with B-trees. The B-tree DDS with FSMs and explicit scheduling give better concurrency than traditional multithreaded straight line code. The B-tree DDS with FSMs and explicit scheduling give better concurrency than traditional multithreaded straight line code. by Barbara Hohlt and Ramakrishna Gummadi

Example User Query : //product[price/msrp<300]/name Unlike current the implementation of Xfilter [AF2000], we only need to retrieve just those profiles which have price < 300 when we have B- tree DDS support. Data (from [AF2000]): Data (from [AF2000]): <catalog> “Color Monitor” “Color Monitor” <msrp>310.40</msrp> </price> Hottest Product Hottest Product </product></catalog>

Key Ideas fsms + queues = reduced locking The use of control-flow driven concurrency control and explicit scheduling permits us to remove locks. fsms + prefetching = reduced blocking “Locking without blocking on I/O” can be enforced more easily by prefetching data and dispatching requests in the order that data becomes available.

Architecture service interacts with DDS via library [library is 2PC coordinator, handles partitioning, replication, etc., and exports B-Tree + HT API] SAN storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” storage “brick” Service (Worker) Service (Worker) DDS lib Service (Worker) Service (Worker) DDS lib Service (Worker) Service (Worker) DDS lib clients interact with any service “front-end” [all persistent state is in DDS and is consistent across cluster] “brick” is durable single-node B-Tree or HT plus RPC skeletons for network access example of a distributed DDS partition with 3 replicas in group Pull Event EventEvent WANPull

PSN Building Blocks RPC (Work) e Push Pull TRANSDUCER Subscribe/Unsubscribe Post e e e e CHANNEL SINK e e1 e2 e3 e FILTER Channel: e1 e2 e3 e FILTER e1 e2 e3 e FILTER Notes: 1) Reliability and delivery semantics embedded within events 2) Events strongly or weakly typed, locally sequenced 3) Channel controls accesses to posts; subscribers use capabilities to unsubscribe 4) As an example, a Channel might notify subscribers when free food is available. The Transducer receives mail events from a mail service and posts food events to the Channel.

Component Layers The application layer makes “search” and “insert” requests to a btree instance. The btree determines what data blocks it needs and fetches them from the global buffer cache. If the cache does not have the needed blocks, it fetches them from the global I/O core, which is transparent to the btree instance. Application Single-Node Btrees Buffer Cache asynchronous I/O Core: “sinks and sources” TCP network VIA network file system storage raw disk storage Distributed Btrees queued completions queued requests

Salient Features of the Implementation Completely lock-free implementation :Completely lock-free implementation : – Possible because of non-blocking FSMs Isolation guaranteed by using queuesIsolation guaranteed by using queues – Invariant: if one request modifies a page ahead of another request, it continues to do so for any other pages that the two happen to access (the second writer always gets the updates of the first) – Sufficient for us because we guarantee A&D only for an action. What happens with transactions?(Delay other requests in queues?) Useful property enforced easily with FSMs and queues: avoid blocking on I/O while holding a lockUseful property enforced easily with FSMs and queues: avoid blocking on I/O while holding a lock – Enforced because we can do explicit scheduling with queues: check if you have what you (mostly) need before you start the FSM, otherwise, pipeline- prefetch (by not blocking) what you may need before you start.

Plan for Success Complete the Single Node B-tree implementationComplete the Single Node B-tree implementation Implement the Distributed B-tree implementationImplement the Distributed B-tree implementation Combine Prof. Franklin’s work with ours to leverage cluster propertiesCombine Prof. Franklin’s work with ours to leverage cluster properties Implement a real-world application…Implement a real-world application…

Conclusions B-trees are good to have as part of DDS for efficiently supporting scalable Publish-Subscribe-Notify (Selective Dissemination of Information) applicationsB-trees are good to have as part of DDS for efficiently supporting scalable Publish-Subscribe-Notify (Selective Dissemination of Information) applications Non-blocking FSMs good for highly-concurrent scalable systems: FSMs give scalability, Non-blocking APIs give concurrencyNon-blocking FSMs good for highly-concurrent scalable systems: FSMs give scalability, Non-blocking APIs give concurrency The DDS API is a useful abstraction on which to easily build cluster applications:The DDS API is a useful abstraction on which to easily build cluster applications: – As proof of concept, we have built a “Food in the Woz” application in less than a week The “Food in the Woz” app makes use of “NinjaMail”, suggesting it is possible to develop reusable components using NinjaThe “Food in the Woz” app makes use of “NinjaMail”, suggesting it is possible to develop reusable components using Ninja