1. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 1 :: Workflows and HPC? :: The relation between workflows and High Performance Computing :: Contact: Lutz Schubert [schubert@hlrs.de], Stefan Wesner [wesner@hlrs.de]

2. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 2 :: The relation between workflows and High Performance Computing No workflows in HPC!

3. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 3 :: The relation between workflows and High Performance Computing No workflows in HPC! Classically, a workflow is – A predefined sequence of tasks / actions – That are exposed in a standard fashion – And communicate via a centralized point Or: “Any program is a workflow” [thanks Mr. Turing]

4. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 4 :: The relation between workflows and High Performance Computing No workflows in HPC! TODO: Picture of a workflow interaction? At the same time: it’s the least efficient way to execute a program: – Centralized communication – Communication overhead through standards – Next steps need to be identified in a central point –…–…  Just the opposite of what HPC programs try to achieve 1 2 3

5. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 5 :: The relation between workflows and High Performance Computing But why should workflows act directly on the process level?  Can also act as a means to control higher-order processes I.Initiate a HPC job as part of a larger process II.Trigger jobs upon specific events (that provide data) III.Coupling jobs according to outcome / status No workflows in HPC?

6. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 6 :: The relation between workflows and High Performance Computing I. Higher-Order Processes with HPC Tasks The “Grid” concept: In particular in engineering cases, extensive calculations need to take place at various stages In large scale, collaborative engineering examples, workflows can reduce the complexity of providing data / jobs A sub-workflow can take over recurring setup and configuration tasks Examples: – Collaborative Engineering (BAE ) – Virtual Engineering (ANSYS)

7. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 7 :: The relation between workflows and High Performance Computing I. Basic Collaborative Structure

8. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 8 :: The relation between workflows and High Performance Computing I. High-Level Workflow

9. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 9 :: The relation between workflows and High Performance Computing II. Event-Based Job Control The “Cloud” Principle: Some recurring jobs need to be executed e.g. with updates in data, under certain environmental conditions etc. These events may trigger recurring configuration workflows (cf. I) Potentially same job offered for multiple users on-demand (with personal data) Not a workflow as such, but easier defined and adapted by making use of workflow descriptions Examples: – Microsoft Financial Computing – Google Market Evaluations

10. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 10 :: The relation between workflows and High Performance Computing Data pro- visioning Job execution Data sub- mission Job sub- mission II. Financial Calculation Principle specific request changes on the global stock market “personal” results triggers

11. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 11 :: The relation between workflows and High Performance Computing III. Coupled Applications Multiple HPC jobs that depend on each other Next job is triggered with the conclusion of the previous one (passing the data) The selection of the next job may depend on the results of the previous ones Intermediary results may trigger evaluation jobs in parallel Examples: – Material Stress Test – Virtual Physiological Human

12. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 12 :: The relation between workflows and High Performance Computing Critical material characteristics Critical points III. Material Stress Test Principle whole modelindividual elements in-depth material test Result feedback Impact on whole model

13. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 13 :: The relation between workflows and High Performance Computing III. A Virtual Physiological Human Model

14. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 14 :: The relation between workflows and High Performance Computing III. A Virtual Physiological Human Model Same base model, but different detailed elements No direct coupling between elements New knowledge about one element can impact on other behaviour Simulation of specific diseases may lead to coupling, depending on goal, e.g. – how medicine spreads if the heart muscle is affected – how a muscular disease spreads to and affects the heart – etc.  Data is only exchanged between elements under certain conditions  The workflow must model these events and conditions

15. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 15 :: The relation between workflows and High Performance Computing To Workflow or Not To Workflow Good For Describing relationships between actors and tasks Recurring configuration and submission processes Result-dependent relationships between jobs Modelling event-based triggers Bad For Tightly coupled process control & execution Distribution of large data between jobs Fast interactions with external processes

16. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 16 :: The relation between workflows and High Performance Computing Summary  Workflows can be used for higher-order control  Tasks in a workflow could be considered as “triggers”  (Large) datasets should be hosted in a well-known place rather than provided via the workflow  Recurring tasks that are not time-critical can be supported by workflows (e.g. configuration steps)  They are more comprehensible than batch jobs and hence easier to model and adapt

17. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 17 :: The relation between workflows and High Performance Computing THANK YOU FOR YOUR ATTENTION - ANY QUESTIONS?

18. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 18 :: The relation between workflows and High Performance Computing Old slides

19. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 19 :: The relation between workflows and High Performance Computing

20. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 20 :: The relation between workflows and High Performance Computing To Workflow or Not To Workflow Workflows can particularly support recurring tasks and typical relationships between jobs / tasks – configuration and submission steps – decision logic – event based triggers They are not sensible to control the distributed nature of a HPC job The stronger the coupling between tasks / elements, the less sensible is workflow support  Higher-order support only

21. :: IDC 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 21 :: The relation between workflows and High Performance Computing III. A Virtual Physiological Human Model