1. 7th Biennial Ptolemy Miniconference Berkeley, CA February 13, 2007 Scheduling Data-Intensive Workflows Tim H. Wong, Daniel Zinn, Bertram Ludäscher (UC Davis)

2. 2Ptolemy Miniconference 2007Daniel Zinn Outline Problem motivation Assumptions Cost model Problem formalization Different “simplifications” and their complexity Prototypical Java implementation for Kepler Summary

3. 3Ptolemy Miniconference 2007Daniel Zinn Motivation: Distributed Execution of Scientific Workflows

4. 4Ptolemy Miniconference 2007Daniel Zinn Motivation: Distributed Execution of Scientific Workflows Process a set of data on a set of machines GOAL: Minimize WF-Execution time! Allocation Problem: Which actors are computed on which hosts?

5. 5Ptolemy Miniconference 2007Daniel Zinn Assumptions Arbitrary data size Arbitrary machine speed Arbitrary bandwidth Arbitrary number of inputs Scientific workflow is a DAG (!) GRID COMPUTING

6. 6Ptolemy Miniconference 2007Daniel Zinn Cost Model Communication Time: T C Function Execution Time: T E Total Time: T T = T C + T E Shipping and Handling Problem: Schedule all tasks such that the total time is minimal

7. 7Ptolemy Miniconference 2007Daniel Zinn Problem Variants and Complexities Task Handling Problem (THP) Data Shipping Problem (DSP) Reduction from Task Scheduling Problem [ERLA94] Reduction from Multiprocessor Scheduling Problem [KA99] Reduction from 1-Multiterminal Cut Shipping and Handling Problem (SHP) Communication Cost: Non-uniform Function Execution Cost: Non-uniform Complexity: NP-complete Communication Cost: Zero Function Execution Cost: Non-uniform Complexity: NP-complete Communication Cost: Non-uniform Function Execution Cost: Zero Complexity: NP-complete

8. 8Ptolemy Miniconference 2007Daniel Zinn easy-DSP: Uniform Transfer Rate, Uniform Data Size Given: Directed Acyclic Graph, Set of Colors Some vertices are already colored Edge Weight = 1, if two adjacent vertices are of different colors Edge Weight = 0, otherwise TASK: Color the rest of the vertices such that total weight is minimal! Cost Model: Minimize Total Shipped Volume! 4

9. 9Ptolemy Miniconference 2007Daniel Zinn 1 - Multi-Terminal CUT Given: Undirected Graph: G = (V,E) Set of Terminals: S V Edge Weights: 1 TASK: Find a multi-way cut of G with a minimum number of edges NP-Complete for more than 3 Terminals! Minimize #edges between different terminals! 4

10. 10Ptolemy Miniconference 2007Daniel Zinn Reduction: 1-MTC <= DSP 4 4 ? DSP 1-MTC “Order graph Color terminals”

11. 11Ptolemy Miniconference 2007Daniel Zinn Reduction: 1-MTC <= DSP 4 4 1 1 1 1 1 1 1 1 1 ?! DSP 1-MTC

12. 12Ptolemy Miniconference 2007Daniel Zinn Reduction: 1-MTC <= DSP 4 4 1 1 1 1 1 1 1 1 1 ! DSP 1-MTC

13. 13Ptolemy Miniconference 2007Daniel Zinn NP-Hard,... But: Need to solve Greedy Algorithm Dynamic Programing Algorithm Investigate Approximation Algorithms for MTC/related !

14. 14Ptolemy Miniconference 2007Daniel Zinn Prototypical Implementation... abstract only some nodes assigned concrete all nodes assigned scheduling

15. 15Ptolemy Miniconference 2007Daniel Zinn Prototypical Implementation... in Kepler! Abstract Workflow... SCHEDULING

16. 16Ptolemy Miniconference 2007Daniel Zinn Prototypical Implementation... in Kepler! Concrete Workflow...

17. 17Ptolemy Miniconference 2007Daniel Zinn Future Work Use Heuristics about looping to guess multiplicities (then not ACYCLIC any more!) Investigate approximation algorithms with error guarantees for 1-MTC => try to apply for DSP ALSO: Relevant for COMAD Workflows: can be “compiled” into a low-level conventional WF

18. 18Ptolemy Miniconference 2007Daniel Zinn Summary Bad news Scheduling is hard DSP is hard (for BEST plans) Good news Finding a quite good plan is easy Greedy/Dynamic Algorithms Open Problems Approximation Quality of “simple algorithms”? When do they perform badly? Does this occur often in real-life workflows?

19. 19Ptolemy Miniconference 2007Daniel Zinn References

20. 20Ptolemy Miniconference 2007Daniel Zinn Thank You. Questions?