1. Introduction to Scalable Programming using Makeflow and Work Queue Dinesh Rajan and Mike Albrecht University of Notre Dame October 24 and November 7, 2012

2. Go to: http://nd.edu/~ccl Click “Tutorial: Introduction to Scalable Programming”

3. 3 I have a standard, debugged, trusted application that runs on my laptop. One simulation runs in an hour. I have to run 100. Then I have to analyze the results, tweak the simulation, and run 100 more. Can I get a single result faster? Can I get more results in the same time?

4. Last year, I heard about this grid thing. This year, I heard about this cloud thing. What do I do next?

5. Should I port my program to MPI or Hadoop? Learn C / Java Learn MPI / Hadoop Re-architect Re-write Re-test Re-debug Re-certify

6. What if my application looks like this?

7. I can get as many machines on the cloud as I want! How do I organize my application to run on those machines?

8. Makeflow: A Portable Workflow System

9. An Old Idea: Makefiles 9 part1 part2 part3: input.data split.py./split.py input.data out1: part1 mysim.exe./mysim.exe part1 >out1 out2: part2 mysim.exe./mysim.exe part2 >out2 out3: part3 mysim.exe./mysim.exe part3 >out3 result: out1 out2 out3 join.py./join.py out1 out2 out3 > result

10. Makeflow = Make + Workflow Provides portability across batch systems. Enable parallelism (but not too much!) Fault tolerance at multiple scales. Data and resource management. 10 Makeflow LocalCondorSGE Work Queue http://www.nd.edu/~ccl/software/makeflow

11. Makeflow Language - Rules Each rule specifies: – a set of target files to create; – a set of source files needed to create them; – a command that generates the target files from the source files. part1 part2 part3: input.data split.py./split.py input.data out1: part1 mysim.exe./mysim.exe part1 >out1 out2: part2 mysim.exe./mysim.exe part2 >out2 out3: part3 mysim.exe./mysim.exe part3 >out3 result: out1 out2 out3 join.py./join.py out1 out2 out3 > result out1 : part1 mysim.exe mysim.exe part1 > out1

12. You must state all the files needed by the command.

13. sims.mf out.10 : in.dat calib.dat sim.exe sim.exe –p 10 in.data > out.10 out.20 : in.dat calib.dat sim.exe sim.exe –p 20 in.data > out.20 out.30 : in.dat calib.dat sim.exe sim.exe –p 30 in.data > out.30

14. Private Cluster Campus Condor Pool Public Cloud Provider CRC SGE Cluster Makefile Makeflow Local Files and Programs Makeflow + Batch System makeflow –T sge makeflow –T condor Work Queue

15. Drivers Local Condor SGE Batch Hadoop WorkQueue Torque MPI-Queue XGrid Moab

16. How to run a Makeflow Run a workflow locally (multicore?) – makeflow -T local sims.mf Clean up the workflow outputs: – makeflow –c sims.mf Run the workflow on SGE: – makeflow –T sge sims.mf

17. Hands On http://nd.edu/~ccl/software/tutorials/ndtut12/mf-tutorial.php

18. Practice Problems http://nd.edu/~ccl/software/tutorials/ndtut12/mf-hw.php 1.Construct a makeflow to render a short movie featuring a Rubik’s cube 2.Launch the makeflow on both your laptop and SGE 3.Consider ways you might use Makeflow for your research