Viktor Prasanna,Yogesh Simmhan, Alok Kumbhare, Sreedhar Natarajan 04/20/2012.

Slides:



Advertisements
Similar presentations
L3S Research Center University of Hanover Germany
Advertisements

A Workflow Engine with Multi-Level Parallelism Supports Qifeng Huang and Yan Huang School of Computer Science Cardiff University
Private and Confidential ThinkControl & ProLiant Essentials Rapid Deployment Pack.
SFIDA-PMI Soluzioni informatiche per FIliere, Distretti ed Associazioni di PMI Genève – 24 Feb 2005 Matteo Villa TXT e-Solutions SpA.
MapReduce.
MapReduce Online Created by: Rajesh Gadipuuri Modified by: Ying Lu.
SmartER Semantic Cloud Sevices Karuna P Joshi University of Maryland, Baltimore County Advisors: Dr. Tim Finin, Dr. Yelena Yesha.
Scalable Content-aware Request Distribution in Cluster-based Network Servers Jianbin Wei 10/4/2001.
Variability Oriented Programming – A programming abstraction for adaptive service orientation Prof. Umesh Bellur Dept. of Computer Science & Engg, IIT.
GridFlow: Workflow Management for Grid Computing Kavita Shinde.
Distributed Systems Architectures
Introduction and Overview “the grid” – a proposed distributed computing infrastructure for advanced science and engineering. Purpose: grid concept is motivated.
Support for Automatic Workflow Composition in Semantic Grid Environemnt Tomasz Gubała, Marian Bubak, Maciej Malawski Institute of Computer Science and.
In-Band Flow Establishment for End-to-End QoS in RDRN Saravanan Radhakrishnan.
Distributed Systems Fall 2011 Gossip and highly available services.
Establishing the overall structure of a software system
Managing Agent Platforms with the Simple Network Management Protocol Brian Remick Thesis Defense June 26, 2015.
The Origin of the VM/370 Time-sharing system Presented by Niranjan Soundararajan.
Špindlerův Mlýn, Czech Republic, SOFSEM Semantically-aided Data-aware Service Workflow Composition Ondrej Habala, Marek Paralič,
WORKFLOW IN MOBILE ENVIRONMENT. WHAT IS WORKFLOW ?  WORKFLOW IS A COLLECTION OF TASKS ORGANIZED TO ACCOMPLISH SOME BUSINESS PROCESS.  EXAMPLE: Patient.
MOBILE CLOUD COMPUTING
資訊工程系智慧型系統實驗室 iLab 南台科技大學 1 Optimizing Cloud MapReduce for Processing Stream Data using Pipelining 出處 : 2011 UKSim 5th European Symposium on Computer Modeling.
Managing Service Metadata as Context The 2005 Istanbul International Computational Science & Engineering Conference (ICCSE2005) Mehmet S. Aktas
Matthew Moccaro Chapter 10 – Deployment and Mobility PART II.
NOVA: CONTINUOUS PIG/HADOOP WORKFLOWS. storage & processing scalable file system e.g. HDFS distributed sorting & hashing e.g. Map-Reduce dataflow programming.
1 A Mutual Exclusion Algorithm for Ad Hoc Mobile networks Presentation by Sanjeev Verma For COEN th Nov, 2003 J. E. Walter, J. L. Welch and N. Vaidya.
CS525: Special Topics in DBs Large-Scale Data Management Hadoop/MapReduce Computing Paradigm Spring 2013 WPI, Mohamed Eltabakh 1.
Hadoop/MapReduce Computing Paradigm 1 Shirish Agale.
Introduction to Hadoop and HDFS
Architecture styles Pipes and filters Object-oriented design Implicit invocation Layering Repositories.
© 2007 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 1 Architectural Styles.
The Limitation of MapReduce: A Probing Case and a Lightweight Solution Zhiqiang Ma Lin Gu Department of Computer Science and Engineering The Hong Kong.
11 CORE Architecture Mauro Bruno, Monica Scannapieco, Carlo Vaccari, Giulia Vaste Antonino Virgillito, Diego Zardetto (Istat)
Clever Framework Name That Doesn’t Violate Copyright Laws MARCH 27, 2015.
Workshop on Future Learning Landscapes: Towards the Convergence of Pervasive and Contextual computing, Global Social Media and Semantic Web in Technology.
Supporting Runtime Reconfiguration on Network Processors Kevin Lee Lancaster University
Grid Computing at Yahoo! Sameer Paranjpye Mahadev Konar Yahoo!
Modeling Component-based Software Systems with UML 2.0 George T. Edwards Jaiganesh Balasubramanian Arvind S. Krishna Vanderbilt University Nashville, TN.
Cloud Age Time to change the programming paradigm?
Optimizing Cloud MapReduce for Processing Stream Data using Pipelining 2011 UKSim 5th European Symposium on Computer Modeling and Simulation Speker : Hong-Ji.
 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.
Mouse BIRN - Jagadeeswaran Rajendiran Pipeline Processing Environment.
Streamflow - Programming Model for Data Streaming in Scientific Workflows Chathura Herath.
EXTENSIBILITY, SAFETY AND PERFORMANCE IN THE SPIN OPERATING SYSTEM
1 Computer Communication & Networks Lecture 21 Network Layer: Delivery, Forwarding, Routing Waleed.
CSC480 Software Engineering Lecture 10 September 25, 2002.
BPEL Business Process Engineering Language A technology used to build programs in SOA architecture.
Abstract A Structured Approach for Modular Design: A Plug and Play Middleware for Sensory Modules, Actuation Platforms, Task Descriptions and Implementations.
CS525: Big Data Analytics MapReduce Computing Paradigm & Apache Hadoop Open Source Fall 2013 Elke A. Rundensteiner 1.
An Overview of Scientific Workflows: Domains & Applications Laboratoire Lorrain de Recherche en Informatique et ses Applications Presented by Khaled Gaaloul.
Aneka Cloud ApplicationPlatform. Introduction Aneka consists of a scalable cloud middleware that can be deployed on top of heterogeneous computing resources.
A N I N - MEMORY F RAMEWORK FOR E XTENDED M AP R EDUCE 2011 Third IEEE International Conference on Coud Computing Technology and Science.
Scheduling MPI Workflow Applications on Computing Grids Juemin Zhang, Waleed Meleis, and David Kaeli Electrical and Computer Engineering Department, Northeastern.
Clever Framework Name MARCH 27, Meeting Agenda  Framework Overview  Prototype 1 Design Goals  Prototype 1 Demo  Prototype 2 Design Goals  Timeline.
Slide 1 Lecture 15 Enterprise Systems Development ( CSC447 ) COMSATS Islamabad Muhammad Usman, Assistant Professor.
Hadoop/MapReduce Computing Paradigm 1 CS525: Special Topics in DBs Large-Scale Data Management Presented By Kelly Technologies
Andy Roberts Data Architect
Is It Possible To Teach Service Science? © Leonard Walletzký PA181 – Service Systems, Modeling and Execution.
Building Custom Workflows
Self Healing and Dynamic Construction Framework:
SOFTWARE DESIGN AND ARCHITECTURE
Complexity Time: 2 Hours.
Microsoft Build /4/2018 9:08 PM Jeff Hollan / Derek Li
Liang Chen Advisor: Gagan Agrawal Computer Science & Engineering
ExaO: Software Defined Data Distribution for Exascale Sciences
Near Real Time ETLs with Azure Serverless Architecture
Jeff Hollan / Derek Li Program Manager – Microsoft
2/19/2019 9:06 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN.
Parallel Algorithm Models
On the Use of Service Level Agreements in AssessGrid
Presentation transcript:

Viktor Prasanna,Yogesh Simmhan, Alok Kumbhare, Sreedhar Natarajan 04/20/2012

Workflow and Stream Processing have been used to for pipeline based applications D3 Science – Dynamic, Distributed, Data Intensive Applications  Dynamism  Data not being static and flowing continuously  Data rates and size being changing depending on domain requirements (QoS requirements) Workflows have compositional characteristics but limit dynamism Stream Processing Systems provide real time processing but lack the compositional and data diversity support Map Reduce framework dynamism in data flow but severely lacks compositional flexibility An architecture which has the capability of  providing Compositional capability and  allows real time stream processing  Provide map reduce based key value exchange Motivation 2

Data Flow Model  Workflows follow Control Flow and data flow  For continuous data, its difficult to define strict control flow  Floe follows a Data Flow Model  Allows for pipelined execution Dynamic Data Mapping Decide whether the output is sent to one output channel(Round Robin) Same Output is sent to every output channel Map Reduce framework wires al Mapper to Reducer Dynamically maps data to reducer at runtime Typed Output Channel Design Paradigms of Floe 3

Continuous Execution System should support continuous processing of data Along with batch processing which takes an input and run once Framework Should be able to pause and resume execution For Low latency applications resources are provisioned and workflow needs to be executed for next batch of input Decentralized Orchestration Centralized Workflow becomes a bottleneck when data flows between tasks which are distributed Decentralized orchestration is better suited, where each component is aware of subsequent component Input Connections, Output Connections etc.. Dynamism in Data Rates & Latency Needs Apart from dynamism in data flow, dynamism occur in data rates and data sizes QoS requirement of Application determines the execution rate by adding new resources at runtime Framework should be able to handle this. Design Paradigms of Floe Contd 4

Elastic Resources  Cloud inherently provides dynamic provisioning of resources  Resources needs to be provisioned ahead of time considering the latency involved in initialization  Application should resilient to overcome the failures Dynamic Task Update  Considering the continuous data flow execution −Pausing, Updating task logic and resuming the workflow in place is costly since the data should be stored −An nice feature would be to have an update tracer event which updates task logic without pausing the workflow Dynamic Data Flow Updates Depending on the requirements structure of a data flow is possible to change Tasks could be added or removed Similar update tracer could be used to update the edge properties rather than the task properties. Design Paradigms of Floe Contd 5

Floe Architecture 6

Smart Grid Streaming Pipeline Use Case 7