WS-VLAM: Towards a Scalable Workflow System on the Grid V. Korkhov, D. Vasyunin, A. Wibisono, V. Guevara-Masis, A. Belloum Institute.

Slides:

Advertisements

Similar presentations

Exploiting Deadline Flexibility in Grid Workflow Rescheduling Wei Chen Alan Fekete Young Choon Lee.

Advertisements

CPSCG: Constructive Platform for Specialized Computing Grid Institute of High Performance Computing Department of Computer Science Tsinghua University.

Barcelona Supercomputing Center. The BSC-CNS objectives: R&D in Computer Sciences, Life Sciences and Earth Sciences. Supercomputing support to external.

A Workflow Engine with Multi-Level Parallelism Supports Qifeng Huang and Yan Huang School of Computer Science Cardiff University

Team involved in Preparing the demo: Presenter: Marcia Inda (SP1.5) Preparing the demo: Adam Belloum (SP2.5), Dmitry Vasunin (SP2.5), Victor Guevara (SP2.5),

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Project: C3-Grid Collaborative Climate Community Data and Processing Grid Scheduling.

GLOBUS PLUG-IN FOR WINGS WOKFLOW ENGINE Elizabeth Martí ITACA Universidad Politécnica de Valencia

WS-VLAM Introduction presentation ws-VLAM workflow Composer System and Network Engineering group Institute of informatics University of Amsterdam.

WS-VLAM Introduction presentation WS-VLAM Workflow Engine System and Network Engineering group Institute of informatics University of Amsterdam.

PARALLEL PROCESSING COMPARATIVE STUDY 1. CONTEXT How to finish a work in short time???? Solution To use quicker worker. Inconvenient: The speed of worker.

WS-VLAM Introduction presentation WS-VLAM Semantic tools Systems, Networking, and Engineering group Institute of informatics University of Amsterdam.

WS-PGRADE: Supporting parameter sweep applications in workflows Péter Kacsuk, Krisztián Karóczkai, Gábor Hermann, Gergely Sipos, and József Kovács MTA.

Lincoln University Canterbury New Zealand Evaluating the Parallel Performance of a Heterogeneous System Elizabeth Post Hendrik Goosen formerly of Department.

2 nd GADA Workshop / OTM 2005 Conferences Eduardo Huedo Rubén S. Montero Ignacio M. Llorente Advanced Computing Laboratory Center for.

SWiM Panel on Engine Implementation Jennifer Widom.

Fault-tolerant Adaptive Divisible Load Scheduling Xuan Lin, Sumanth J. V. Acknowledge: a few slides of DLT are from Thomas Robertazzi ’ s presentation.

1 Introduction to Load Balancing: l Definition of Distributed systems. Collection of independent loosely coupled computing resources. l Load Balancing.

Workflow Management System based on Service Oriented Components for Grid Applications. Ju-Ho Choi Korea University, Seoul, Rep. of Korea.

UvA, Amsterdam June 2007WS-VLAM Introduction presentation WS-VLAM Requirements list known as the WS-VLAM wishlist System and Network Engineering group.

The new The new MONARC Simulation Framework Iosif Legrand  California Institute of Technology.

WS-VLAM Introduction presentation WS-VLAM Introduction Systems and Network Engineering group Institute of informatics University of Amsterdam.

GHS: A Performance Prediction and Task Scheduling System for Grid Computing Xian-He Sun Department of Computer Science Illinois Institute of Technology.

Workload Management Massimo Sgaravatto INFN Padova.

- 1 - Grid Programming Environment (GPE) Ralf Ratering Intel Parallel and Distributed Solutions Division (PDSD)

An approach for solving the Helmholtz Equation on heterogeneous platforms An approach for solving the Helmholtz Equation on heterogeneous platforms G.

 Escalonamento e Migração de Recursos e Balanceamento de carga Carlos Ferrão Lopes nº M6935 Bruno Simões nº M6082 Celina Alexandre nº M6807.

DISTRIBUTED COMPUTING

WP9 Resource Management Current status and plans for future Juliusz Pukacki Krzysztof Kurowski Poznan Supercomputing.

An Autonomic Framework in Cloud Environment Jiedan Zhu Advisor: Prof. Gagan Agrawal.

Trace Generation to Simulate Large Scale Distributed Application Olivier Dalle, Emiio P. ManciniMar. 8th, 2012.

A Novel Approach to Workflow Management in Grid Environments Frank Berretz*, Sascha Skorupa*, Volker Sander*, Adam Belloum** 15/04/2010 * FH Aachen - University.

The ACGT Workflow Editing & Enactment Environment Giorgos Zacharioudakis Institute of Computer Science, Foundation for Research & Technology – Hellas (ICS-FORTH)

MARISSA: MApReduce Implementation for Streaming Science Applications 作者 : Fadika, Z. ; Hartog, J. ; Govindaraju, M. ; Ramakrishnan, L. ; Gunter, D. ; Canon,

© DATAMAT S.p.A. – Giuseppe Avellino, Stefano Beco, Barbara Cantalupo, Andrea Cavallini A Semantic Workflow Authoring Tool for Programming Grids.

April 26, CSE8380 Parallel and Distributed Processing Presentation Hong Yue Department of Computer Science & Engineering Southern Methodist University.

Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, October 2003.

A Hierarchical MapReduce Framework Yuan Luo and Beth Plale School of Informatics and Computing, Indiana University Data To Insight Center, Indiana University.

Service - Oriented Middleware for Distributed Data Mining on the Grid ，劉妘鑏 Antonio C., Domenico T., and Paolo T. Journal of Parallel and Distributed.

Autonomic scheduling of tasks from data parallel patterns to CPU/GPU core mixes Published in: High Performance Computing and Simulation (HPCS), 2013 International.

ServiceSs, a new programming model for the Cloud Daniele Lezzi, Rosa M. Badia, Jorge Ejarque, Raul Sirvent, Enric Tejedor Grid Computing and Clusters Group.

Parallelizing Video Transcoding Using Map-Reduce-Based Cloud Computing Speaker : 童耀民 MA1G0222 Feng Lao, Xinggong Zhang and Zongming Guo Institute of Computer.

O PTIMAL SERVICE TASK PARTITION AND DISTRIBUTION IN GRID SYSTEM WITH STAR TOPOLOGY G REGORY L EVITIN, Y UAN -S HUN D AI Adviser: Frank, Yeong-Sung Lin.

Interactive Workflows Branislav Šimo, Ondrej Habala, Ladislav Hluchý Institute of Informatics, Slovak Academy of Sciences.

WS-VLAM Tutorial Part I: Hands on the User Graphical Interface Adam Belloum.

Distributed Computing With Triana A Short Course Matthew Shields, Ian Taylor & Ian Wang.

George Goulas, Christos Gogos, Panayiotis Alefragis, Efthymios Housos Computer Systems Laboratory, Electrical & Computer Engineering Dept., University.

Enabling Self-management of Component-based High-performance Scientific Applications Hua (Maria) Liu and Manish Parashar The Applied Software Systems Laboratory.

Department of Computer Science MapReduce for the Cell B. E. Architecture Marc de Kruijf University of Wisconsin−Madison Advised by Professor Sankaralingam.

A Hyper-heuristic for scheduling independent jobs in Computational Grids Author: Juan Antonio Gonzalez Sanchez Coauthors: Maria Serna and Fatos Xhafa.

Support for cooperative experiments in VL-e: from scientific workflows to knowledge sharing.

A Grid-enabled Multi-server Network Game Architecture Tianqi Wang, Cho-Li Wang, Francis C.M.Lau Department of Computer Science and Information Systems.

International Symposium on Grid Computing (ISGC-07), Taipei - March 26-29, 2007 Of 16 1 A Novel Grid Resource Broker Cum Meta Scheduler - Asvija B System.

Xi He Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY THERMAL-AWARE RESOURCE.

Evaluating Meta-Scheduling Algorithms in VLAM-G Environment V.Korkhov, A.Belloum, L.O.Hertzberger FNWI, University of Amsterdam Key VLAM-G applications.

Virtual Lab AMsterdam VLAMsterdam Abstract Machine Toolbox A.S.Z. Belloum, Z.W. Hendrikse, E.C. Kaletas, H. Afsarmanesh and L.O. Hertzberger Computer Architecture.

Hierarchical Load Balancing for Large Scale Supercomputers Gengbin Zheng Charm++ Workshop 2010 Parallel Programming Lab, UIUC 1Charm++ Workshop 2010.

Compilation of XSLT into Dataflow Graphs for Web Service Composition Peter Kelly Paul Coddington Andrew Wendelborn.

WS-PGRADE/gUSE in use Advance use of WS- PGRADE/gUSE gateway framework Zoltán Farkas and Peter Kacsuk MTA SZTAKI LPDS.

Collection and storage of provenance data Jakub Wach Master of Science Thesis Faculty of Electrical Engineering, Automatics, Computer Science and Electronics.

December, 2006 ws-VLAM Workflow Management System a Re-factoring of VLAM Dmitry Vasyunin Adianto Wibisono Adam Belloum.

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.

Accelerating K-Means Clustering with Parallel Implementations and GPU Computing Janki Bhimani Miriam Leeser Ningfang Mi

INTRODUCTION TO HIGH PERFORMANCE COMPUTING AND TERMINOLOGY.

| presented by Vasileios Zois CS at USC 09/20/2013 Introducing Scalability into Smart Grid 1.

Introduction to Load Balancing:

Conception of parallel algorithms

Recap: introduction to e-science

CSE8380 Parallel and Distributed Processing Presentation

GGF10 Workflow Workshop Summary

Presentation transcript:

WS-VLAM: Towards a Scalable Workflow System on the Grid V. Korkhov, D. Vasyunin, A. Wibisono, V. Guevara-Masis, A. Belloum Institute of informatics Faculty of Science University of Amsterdam

Outline Introduction: what is WS-VLAM? Architecture of the WS-VLAM Large-scale workflow support:  Distributed workflow engine and multi-cluster execution support  Hierarchical resource management and workload balancing  Workflow farming  Semantic workflow support Conclusions

Introduction WS-VLAM (Virtual Lab AMsterdam) concepts: Data driven workflow system Data streaming between workflow components running on the Grid Components: input and output ports for data exchange; parameters for control (during runtime as well); graphical output (X11) supported GUI and engine decoupled, interfaced using WS-RF Engine (RTS – Run Time System): Implemented as GT4 WS-RF service Uses GT4 features (delegation service, GSI, notifications etc.)

WS-VLAM architecture

Large-scale distributed workflows support Multi-cluster distributed experiments: distributed workflow engine Heterogeneous resources: workload balancing and resource management Complex workflows with parameter sweeps and iterative processing: workflow farming Semantic support

Distributed workflow engine GT4 Service Container WS-RTSM Factory EPR GRAM WS-RTSM Instance Worker nodes Workflow components Workflow components GRAM Worker nodes Workflow components Workflow components GT4 Service Container WS-RTSM Instance GUI proxy Data proxy Data proxy GUI proxy Distributed RTSM WS-RTSM Factory Distributed RTSM Cluster 1Cluster 2 WS-VLAM GUI Resource Manager

Hierarchical resource management and workload balancing Task level: Adaptive workload balancing for parallel applications (MPI) on heterogeneous resources Job level: inter-task workload distribution and balancing for multi-task applications (DIANE user-level scheduling env.) Workflow level: workflow farming

Workload balancing strategy (parallel and multi-task applications) Distribution of divisible workload between tasks based on application characteristics (communications/computations ratio) and resource characteristics (CPU, memory, bw) Weights are assigned to all the resources that execute tasks according to their capacities Fast heuristic algorithm for approximate weighting of resources processing the workload Iterative processing of similar data; measuring execution performance for each iteration and adapting weights (and thus workload distribution) on the fly

Workflow farming: adaptive data distribution WF WF 1 WF 2 EstimatorDistributor Each farmed workflow gets a single data element to process first to assess its performance. The speed of processing is evaluated, then the future workload distribution is determined according to this information. Weights reflecting the performance are assigned to the workflows. WF 1 is twice as slow! W=1 W=2 Iterative processing: Independent data or parameters

WF 1 WS-RTSM 1 WF 2 WS-RTSM 2 WF 3 WS-RTSM 3 Resource Manager Workflows WF1,2,3 are running, having WS interface, ready to process data from the RM “on-demand” RTSM Factory XML topology Data to farm Perf Workflow farming: WF service List of WS-RTSM EPRs Performance data GUI

Semantic workflow support

Conclusions WS-VLAM features towards large scale data driven workflows support:  Multi-cluster support for a single workflow, ability for data exchange between internal nodes of different clusters  Adaptive workload balancing for parallel applications (workflow components) on heterogeneous resources  Workload balancing on workflow level: parameter/data sweep for workflow  Semantic support for workflow composition