1 Trace-Based Characteristics of Grid Workflows Alexandru Iosup and Dick Epema PDS Group Delft University of Technology The Netherlands Simon Ostermann,

Slides:



Advertisements
Similar presentations
Challenge the future Delft University of Technology Overprovisioning for Performance Consistency in Grids Nezih Yigitbasi and Dick Epema Parallel.
Advertisements

28 October 2010 Challenge the future Delft University of Technology Cost-driven Scheduling of Grid Workflows Using Partial Critical Paths Dick Epema Delft.
INSPIRE The Insieme Parallel Intermediate Representation Herbert Jordan, Peter Thoman, Simone Pellegrini, Klaus Kofler, and Thomas Fahringer University.
Scheduling in Distributed Systems Gurmeet Singh CS 599 Lecture.
Evaluating Heuristics for the Fixed-Predecessor Subproblem of Pm | prec, p j = 1 | C max.
Tai, Yu-Chang 4/29/2013 Future Generation Computer Systems(FGCS.J) journal homepage: Saeid Abrishami a, ∗, Mahmoud Naghibzadeha,
Managing Risk of Inaccurate Runtime Estimates for Deadline Constrained Job Admission Control in Clusters Chee Shin Yeo and Rajkumar Buyya Grid Computing.
1 Sensor Relocation in Mobile Sensor Networks Guiling Wang, Guohong Cao, Tom La Porta, and Wensheng Zhang Department of Computer Science & Engineering.
June 1, Inter-Operating Grids through Delegated MatchMaking Alexandru Iosup, Dick Epema PDS Group, TU Delft, NL Todd Tannenbaum, Matt Farrellee,
Trace-Based Automatic Parallelization in the Jikes RVM Borys Bradel University of Toronto.
June 2, GrenchMark : A Framework for Analyzing, Testing, and Comparing Grids CCGrid 2006 A. Iosup, D.H.J. Epema PDS Group, ST/EWI, TU Delft.
June 3, ServMark A Hierarchical Architecture for Testing Grids Santiago, Chile A. Iosup, H. Mohamed, D.H.J. Epema PDS Group, ST/EWI, TU Delft C.
June 3, 2015 Synthetic Grid Workloads with Ibis, K OALA, and GrenchMark CoreGRID Integration Workshop, Pisa A. Iosup, D.H.J. Epema Jason Maassen, Rob van.
The Performance of Bags-Of-Tasks in Large-Scale Distributed Computing Systems Alexandru Iosup, Ozan Sonmez, Shanny Anoep, and Dick Epema ACM/IEEE Int’l.
Inter-Operating Grids through Delegated MatchMaking Alexandru Iosup, Dick Epema, Hashim Mohamed,Mathieu Jan, Ozan Sonmez 3 rd Grid Initiative Summer School,
1 Google Workshop at TU Delft, 2010 – Online Games and Clouds Cloudifying Games: Rain for the Thirsty Alexandru Iosup Parallel and Distributed Systems.
1 A Performance Study of Grid Workflow Engines Alexandru Iosup and Dick Epema PDS Group Delft University of Technology The Netherlands Corina Stratan Parallel.
Fault-tolerant Adaptive Divisible Load Scheduling Xuan Lin, Sumanth J. V. Acknowledge: a few slides of DLT are from Thomas Robertazzi ’ s presentation.
June 25, GrenchMark: A synthetic workload generator for Grids KOALA Workshop A. Iosup, H. Mohamed, D.H.J. Epema PDS Group, ST/EWI, TU Delft.
June 25, GrenchMark: Synthetic workloads for Grids First Demo at TU Delft A. Iosup, D.H.J. Epema PDS Group, ST/EWI, TU Delft.
GHS: A Performance Prediction and Task Scheduling System for Grid Computing Xian-He Sun Department of Computer Science Illinois Institute of Technology.
June 28, Resource and Test Management in Grids Rapid Prototyping in e-Science VL-e Workshop, Amsterdam, NL Dick Epema, Catalin Dumitrescu, Hashim.
July 13, “How are Real Grids Used?” The Analysis of Four Grid Traces and Its Implications IEEE Grid 2006 Alexandru Iosup, Catalin Dumitrescu, and.
Euro-Par 2008, Las Palmas, 27 August DGSim : Comparing Grid Resource Management Architectures Through Trace-Based Simulation Alexandru Iosup, Ozan.
Copyright © , Software Engineering Research. All rights reserved. Creating Responsive Scalable Software Systems Dr. Lloyd G. Williams Software.
1 Efficient Management of Data Center Resources for Massively Multiplayer Online Games V. Nae, A. Iosup, S. Podlipnig, R. Prodan, D. Epema, T. Fahringer,
August 28, Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing Berkeley, CA, USA Alexandru Iosup, Nezih Yigitbasi,
Euro-Par 2007, Rennes, 29th August 1 The Characteristics and Performance of Groups of Jobs in Grids Alexandru Iosup, Mathieu Jan *, Ozan Sonmez and Dick.
Graphical Analysis. Why Graph Data? Graphical methods Require very little training Easy to use Massive amounts of data can be presented more readily Can.
LDBC-Benchmarking Graph-Processing Platforms: A Vision Benchmarking Graph-Processing Platforms: A Vision (A SPEC Research Group Process) Delft University.
1 Cloud Computing Research at TU Delft – A. Iosup Alexandru Iosup Parallel and Distributed Systems Group Delft University of Technology The Netherlands.
A Performance Evaluation of Azure and Nimbus Clouds for Scientific Applications Radu Tudoran KerData Team Inria Rennes ENS Cachan 10 April 2012 Joint work.
1 EuroPar 2009 – POGGI: Puzzle-Based Online Games on Grid Infrastructures POGGI: Puzzle-Based Online Games on Grid Infrastructures Alexandru Iosup Parallel.
Computer Science Department University of Pittsburgh 1 Evaluating a DVS Scheme for Real-Time Embedded Systems Ruibin Xu, Daniel Mossé and Rami Melhem.
Young Suk Moon Chair: Dr. Hans-Peter Bischof Reader: Dr. Gregor von Laszewski Observer: Dr. Minseok Kwon 1.
Jockey Guaranteed Job Latency in Data Parallel Clusters Andrew Ferguson, Peter Bodik, Srikanth Kandula, Eric Boutin, and Rodrigo Fonseca.
Xiao Liu CS3 -- Centre for Complex Software Systems and Services Swinburne University of Technology, Australia Key Research Issues in.
Xiao Liu, Jinjun Chen, Ke Liu, Yun Yang CS3: Centre for Complex Software Systems and Services Swinburne University of Technology, Melbourne, Australia.
“WHY ARE PROJECTS ALWAYS LATE?” (“and what can the Project Manager DO about that?) Craig Henderson, MBA, PMP ARVEST Bank Operations.
Supercomputing Cross-Platform Performance Prediction Using Partial Execution Leo T. Yang Xiaosong Ma* Frank Mueller Department of Computer Science.
1 Challenge the future KOALA-C: A Task Allocator for Integrated Multicluster and Multicloud Environments Presenter: Lipu Fei Authors: Lipu Fei, Bogdan.
Euro-Par, A Resource Allocation Approach for Supporting Time-Critical Applications in Grid Environments Qian Zhu and Gagan Agrawal Department of.
1 ROIA 2009 – CAMEO: Continuous Analytics for Massively Multiplayer Online Games CAMEO: Continuous Analytics for Massively Multiplayer Online Games Alexandru.
Stochastic DAG Scheduling using Monte Carlo Approach Heterogeneous Computing Workshop (at IPDPS) 2012 Extended version: Elsevier JPDC (accepted July 2013,
Predicting Queue Waiting Time in Batch Controlled Systems Rich Wolski, Dan Nurmi, John Brevik, Graziano Obertelli Computer Science Department University.
Enabling Grids for E- sciencE EGEE and gLite are registered trademarks EGEE-III INFSO-RI Analysis of Overhead and waiting times.
1 Test Selection for Result Inspection via Mining Predicate Rules Wujie Zheng
Understanding Performance, Power and Energy Behavior in Asymmetric Processors Nagesh B Lakshminarayana Hyesoon Kim School of Computer Science Georgia Institute.
A Hyper-heuristic for scheduling independent jobs in Computational Grids Author: Juan Antonio Gonzalez Sanchez Coauthors: Maria Serna and Fatos Xhafa.
1/22 Optimization of Google Cloud Task Processing with Checkpoint-Restart Mechanism Speaker: Sheng Di Coauthors: Yves Robert, Frédéric Vivien, Derrick.
OPERATING SYSTEMS CS 3530 Summer 2014 Systems and Models Chapter 03.
Xi He Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY THERMAL-AWARE RESOURCE.
Introducing Α UML model For Faster-than-Real-Time Simulation Dimosthenis Anagnostopoulos 1, Vassilis Dalakas 2, George- Dimitrios Kapos 1, Mara Nikolaidou.
Ensieea Rizwani An energy-efficient management mechanism for large-scale server clusters By: Zhenghua Xue, Dong, Ma, Fan, Mei 1.
A stochastic scheduling algorithm for precedence constrained tasks on Grid Future Generation Computer Systems (2011) Xiaoyong Tang, Kenli Li, Guiping Liao,
Introduction to SHIWA project EGI User Forum, Vilnius Peter Kacsuk MTA SZTAKI
1 Performance Impact of Resource Provisioning on Workflows Gurmeet Singh, Carl Kesselman and Ewa Deelman Information Science Institute University of Southern.
INTRODUCTION TO XSEDE. INTRODUCTION  Extreme Science and Engineering Discovery Environment (XSEDE)  “most advanced, powerful, and robust collection.
Distributed Correlation in Fabric Kiwi Team PSNC.
OPERATING SYSTEMS CS 3502 Fall 2017
Introduction | Model | Solution | Evaluation
CHAPTER 8 Operations Scheduling
A Dynamic Critical Path Algorithm for Scheduling Scientific Workflow Applications on Global Grids e-Science IEEE 2007 Report: Wei-Cheng Lee
Namyoon Woo and Heon Y. Yeom
Overview: Fault Diagnosis
Department of Computer Science & Engineering
On Dynamic Resource Availability in Grids
An Adaptive Middleware for Supporting Time-Critical Event Response
ANALYSIS OF USER SUBMISSION BEHAVIOR ON HPC AND HTC
Model Transformation with the Ptera Controller
Presentation transcript:

1 Trace-Based Characteristics of Grid Workflows Alexandru Iosup and Dick Epema PDS Group Delft University of Technology The Netherlands Simon Ostermann, Radu Prodan, and Thomas Fahringer DPS Group University of Innsbruck Austria

2 Why are Grid Workflows Interesting? Grids promise reliable and easy-to-use computational infrastructure for e-Science Full automation from experiment design to final result Often, automation = workflows Jobs comprising inter-related computing and data-transfer tasks

3 Why are the Characteristics of Grid Workflows Interesting? For focusing on the right research problems What are the interesting characteristics? Number of nodes? Number of edges? Other characteristics… For simulation studies Optimizing a scheduler for one workload does not make it useful for another (often quite the contrary) … optimizing for a workload type is better For performance evaluation in real environments The system tuned to one workload

4 Outline Introduction Method for Grid Workflow Analysis Austrian Grid Traces Grid Workflow Characteristics Conclusion and Future Work

5 Method for Grid Workflow Analysis [1/3] Overview Goal: establish the main characteristics of grid workflow such that building a workflow-based grid workload model is greatly facilitated Grid workflow characteristics Workflow-intrinsic Environment-related

6 Method for Grid Workflow Analysis [2/3] Intrinsic Workflow Characteristics Size and structure of the workflow Number of nodes (N)/edges (E) Branching Factor = N/E Work Size = task runtime of a task on a base platform [SI2k] Work Size Variability = ratio longest vs. shortest WF task Sequential execution path Critical execution path Graph level (L) = length of critical execution path Arrival patterns Daily patterns: Peak Hours Weekly patterns: Week-end vs. Work Days

7 Method for Grid Workflow Analysis [3/3] Environment-Related WF Characteristics Time-related Makespan (MS) = time between WF entering and exiting system Scheduler-related Speedup (S) = MS / Sequential Execution Path Size Normalized Schedule Length (NSL) = MS / Critical Path Size Failure-related Success rate = % tasks finished correctly, per WF

8 Outline Introduction Method for Grid Workflow Analysis Austrian Grid Traces Grid Workflow Characteristics Conclusion and Future Work

9 The Austrian Grid Traces Austrian Grid: 8 sites, ~500 processors Two non-overlapping long-term traces from two workflow engines: Askalon DEE, Askalon EE2 Workflows: mostly testing, but many jobs similar to production workflows Production areas: material sciences, astrophysics, weather prediction, engineering, movie rendering

10 Outline Introduction Method for Grid Workflow Analysis Austrian Grid Traces Grid Workflow Characteristics Conclusion and Future Work

11 Intrinsic Workflow Characteristics [1/3] Number of nodes 75% WFs have <40 tasks 95% WFs have < 200 tasks 200 tasks 40 tasks

12 Intrinsic Workflow Characteristics [2/3] Task Work Size >80% WFs take <2 minutes on 1000-SI2k machine >95% WFs take <10 minutes on 1000-SI2k machine 10 mins 2 mins

13 Intrinsic Workflow Characteristics [3/3] Work Size Variability >80% WFs have <10 task size variability But still many WFs with high task size variability

14 Classes of Workflows Simple classifier (experience from previous work) Future: data mining techniques

15 Environment-Related Characteristics Workflow class matters: better SU for “easier” classes Large-and-Flat “easier” than Large-and-Branchy Large-and-Branchy “easier” than Branchy (o/head)

16 Outline Introduction Method for Grid Workflow Analysis Austrian Grid Traces Grid Workflow Characteristics Conclusion and Future Work

17 Conclusion and Future Work Method for the analysis of grid workflows Intrinsic workflow characteristics Environment-dependent workflow characteristics More statistical details than average/std.deviation (Normal is not the typical distribution in computer science) Analysis of two workflow-based traces from Austrian Grid Future work Apply method to more traces Design workflow-based grid workload model

18 Thank you! Questions? Remarks? Observations? Help building our community’s Grid Workloads Archive:

Feedback: Privacy Policy Feedback
Do Not Sell My Personal Information
About project: SlidePlayer Terms of Service

© 2024 SlidePlayer.com. Inc. All rights reserved.