Scientific Applications with HyperFlow and Scalarm on the PaaSage Platform Marian Bubak Department of Computer Science AGH University of Science and Technology Krakow, Poland http://dice.cyfronet.pl ESOCC 2016 Conference, Vienna, September 5-7, 2016

Coauthors Maciej Malawski Bartosz Balis Kamil Figiela Maciej Pawlik Dariusz Krol Renata Slota Michal Orzechowski Jacek Kitowski Dennis Hoppe dice.cyfronet.pl www.hlrs.de

Outline Motivation: scientific applications vs cloud resources Cloud platform – PaaSage Application deployment and execution modeling with CAMEL Workflows on clouds with HyperFlow Scalability and scheduling of workflows Parameter study on clouds with Scalarm Sample results Conclusions

Scientific applications vs cloud resources Cloud = a complex ecosystem Users, advanced middleware services, security requirements, resource usage quotas, etc. Challenge: a solution for deployment and execution of scientific applications on clouds in an automated cost- and performance-effective way: adaptable to diverse cloud infrastructures transparent to the user lightweight: minimizes user’s effort (setup, configuration) maintainable: easy to integrate and fast to update leverage cloud elasticity for autoscaling of scientific workflows loosely coupled integration with cloud management platforms deployment description in a cloud independent way automated deployment in cross-cloud environments cost- and performance-based deployment optimization scaling out/in based on application-specific metrics Maciej Malawski, Bartosz Balis, Kamil Figiela, Maciej Pawlik, Marian Bubak: Support for Scientific Workflows in a Model-Based Cloud Platform. UCC 2015: 412-413

HLRS Molecular dynamics Docking task is a parallel MPI job either a 32+ core VM Or multiple VMs (virtual cluster) Compute intensive workflow e.g. on 16 cores, 8M molecules, simulation time 0.05: Run time 50 minutes Initialization Preprocessing MD simuation Postprocessing End

Montage (astronomy) workflow Characteristics: all tasks are "dataflow" all consume and produce files each task is fired only once tasks are I/O intensive Execution model: HyperFlow passes the command to be invoked on a VM for each task Executor fetches tasks ready for execution from a queue, executes the command, and puts a message back on the queue Very large (100K+) pipelines (DAGs) of resource-intensive tasks

eScience workflows on the PaaSage platform PaaSage community e.g. MD from HLRS PL-Grid community: bioinformatics (genomics, proteomics) metals engineering (complex metallurgical processes) Virtual Physiological Human (Taverna and DataFluo workflows) multiscale applications: fusion (Kepler workflows) military mission planning support (EDA) astronomy (Pegasus workflows) Results HyperFlow: workflow execution engine based on REST paradigm Scalarm: massively self-scalable platform for data farming

PaaSage Platform http://www.paasage.eu An open and integrated platform with an accompanying methodology that allows model-based development, configuration, optimisation, and deployment of cloud applications independently of underlying cloud infrastructures. http://www.paasage.eu

Model-based development and deployment of cloud applications PaaSage cloud platform: CAMEL: Cloud Application Modeling and Execution Language Deployment model: components, connections Requirements model Scalability model Multi-cloud application deployment Autoscaling, adaptation Integration with workflow systems: CAMEL app model is generated from a workflow description Workflow monitoring information triggers workflow autoscaling

Scientific applications on multi-clouds Extensions to PaaSage platform a new Domain-Specific Language for describing workflows workflow planning based on user-provided objectives and constraints a new workflow execution engine Take benefit of PaaSage services automatic and adaptive deployment of workflow components autoscaling driven by rules Workflow description (DAG) Reasoner Prepares a deployment plan Workflow Engine Executes workflow tasks Deployer Launches VMs Constraint, objectives Executionware Deployment plan, autoscaling rules Enforcement engine Applies rules Adapter Concrete deployment commands and rules Monitoring events Upperware Workflow Planner Plans workflow execution CAMEL description of workflow application Bartosz Balis, Kamil Figiela, Maciej Malawski, Maciej Pawlik, Marian Bubak: A Lightweight Approach for Deployment of Scientific Workflows in Cloud Infrastructures. PPAM (1) 2015: 281-290

Lightweight workflow programming and execution environment Simple wf description (JSON) Advanced programming of wf activities (JavaScript) Running a workflow – simple command line client hflowc run <workflow_dir> function getPathWayByGene(ins, outs, config, cb) { var geneId = ins.geneId.data[0], url = ... http({"timeout": 10000, "url": url }, function(error, response, body) { ... cb(null, outs); }); } <workflow_dir> contains: File workflow.json (wf graph) File workflow.cfg (wf config) Optionally: file functions.js (advanced workflow activities) Input files

HyperFlow on the PaaSage platform PaaSage cloud platform: Model-based development of cloud applications Multi-cloud application deployment Autoscaling according to application demands Integration of HyperFlow with PaaSage: CAMEL app model generated from HyperFlow Wf description Task scheduler delivers initial deployment plan and scalability rules Workflow monitoring information triggers workflow autoscaling Bartosz Baliś, Marian Bubak, Kamil Figiela, Maciej Malawski, Maciej Pawlik, Towards Deployment and Autoscaling of Scientific Workflows with HyperFlow and PaaSage, CGW’14

Workflow generic scenario: HLRS MD with MPI Worker Storage Workflow Engine (Master) MPI Worker MPI Master MPI App Bin Job Exec WWW NFS Rabbit HyperFlow Redis Worker MPI Worker MPI Master Flexiant [location: UK] Each HyperFlow worker requires a set of machines - a virtual cluster consisting of MPI master and MPI worker The VMs need to be on the same IP subdomain Application Binaries App Bin RabbitMQ Server Rabbit Job Executor Job Exec HyperFlow Engine HyperFlow NFS Server NFS Redis DB Redis MPI Runtime MPI WWW Web server 13

Example application goals and scalability rules My workflow can run in parallel and can scale up to 8 VMs I need to dynamically scale out my virtual cluster based on utilization: if resource utilization of VMs > 90% for more than 3 minutes then add a new worker VM if resource utilization of VMs < 10% for more than 3 minutes then terminate this worker VM I prepared an execution plan with constraints: My workflow consists of 2 stages: For stage 1 I need 8 VMs; For stage 2 I need to add 8 more VMs → please monitor the WorkflowStage metric published by workflow engine and if WorkflowStage > 1 then add 8 worker VMs All VMs need to be of the same type: m3.xxxlarge on Amazon or 8-core VM on any provider (alternative) I need 16 core-hours to run my workflow Please find the cheapest deployment Additional constraint: I have a quota on VM number: Max 8 instances on OpenStack Max 20 instances on Amazon I need to terminate all workers when the workflow is complete → please monitor WorklfowExecutionState metric published by workflow engine and if WorkflowExecutionState == DONE then terminate all the workers

Scheduling and provisioning plan Example workflow with 3 stages Scaling rule: launch 7 VMs for stage 2 Scaling rule: terminate 2 VMs for stage 3 Tasks of stage 1 VM Tasks of stage 2 Tasks of stage 3 Time Maciej Malawski, Kamil Figiela, Marian Bubak, Ewa Deelman, Jarek Nabrzyski: Scheduling Multilevel Deadline-Constrained Scientific Workflows on Clouds Based on Cost Optimization. Scientific Programming 2015: 680271:1-680271:13 (2015)

Parameter study with Scalarm on PaaSage platform Scalarm orchestrates parameter studies and data farming experiments PaaSage framework manages Scalarm deployment in cross-cloud environment Scalarm is modeled (services, deployment and scaling) with the CAMEL language Obtained solution is published in the PaaSage social network D. Król, M. Orzechowski, J. Liput, R. Słota, and J. Kitowski. Model-based execution of scientific applications on cloud infrastructures: Scalarm case study, in: Proceedings of Cracow Grid Workshop, pp. 77-78, 2014. D. Król, R. Da Silva, E. Deelman and V. Lynch, Workflow Performance Profiles: Development and Analysis, accepted: The International Workshop on Algorithms, Models and Tools for Parallel Computing on Heterogeneous Platforms (HeteroPar'2016), Euro-Par 2016.

Main goal: execution of the same application with different input parameter values i.e. support different steps of a data farming/parameter studies process input parameter space specification application execution with different input parameter values collecting results and analysis

Overview of results Behavior analysis of security forces M. Kvassay, L. Hluchý, S. Dlugolinský, B. Schneider, H. Bracker, A. Tavčar, M. Gams, M. Contat, L. Dutka, D. Król, M. Wrzeszcz, J. Kitowski, A Novel Way of Using Simulations to Support Urban Security Operations. COMPUTING AND INFORMATICS, 34(6), 2015. Molecular dynamics - nano droplet simulation D. Król, M. Orzechowski, J.Kitowski, Ch. Niethammer, A. Sulisto, A. Wafai, A Cloud-Based Data Farming Platform for Molecular Dynamics Simulations, in: proc. 7th IEEE/ACM International Conference on Utility and Cloud Computing (UCC), 8-11, London UK, IEEE 2014, pp. 579 – 584. Hot rolling mill design D. Król, R. Slota, J. Kitowski, L. Rauch, K. Bzowski, M. Pietrzyk, Model-based approach to study hot rolling mills with data farming, in: T. Claus, et al. (eds.), Proc. of 30th European Conf. on Modelling and Simulations, Regensburg, 2016, OTH Regensburg 2016, pp. 495-501. Sensitivity analysis D. Bachniak, J. Liput, L. Rauch, R. Słota, and J. Kitowski. Massively Parallel Approach to Sensitivity Analysis on HPC Architectures by using Scalarm Platform. In Parallel Processing and Applied Mathematics : 11th international conference, PPAM 2015 : Kraków, Poland, September 6–9, 2015 : book of abstracts,, page 93, 2015. Material science - molecular dynamics + neutron scattering intensity calculations D. Król, R. Silva, E. Deelman, V. E. Lynch, Workflow Performance Profiles: Development and Analysis, accepted at Euro-Par 2016 (HeteroPar’16).

Summary HyperFlow + PaaSage improve reproducibility: CAMEL: complete description of infrastructure HyperFlow JSON: complete description of application On-demand deployment of the workflow runtime environment as part of the workflow application Workflow engine as another app component driving the execution of other components Avoidance of tight coupling to a particular cloud infrastructure and middleware

https://github.com/dice-cyfronet/hyperflow More at http://www.paasage.eu https://github.com/dice-cyfronet/hyperflow http://scalarm.com http://dice.cyfronet.pl bubak@agh.edu.pl

Acknowledgements This research was supported by EU FP-7 ICT Project PaaSage – 317715 Polish Grant 3033/7PR/2014/2