P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G. Kecskemeti and G. Hermann P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G. Kecskemeti and G. Hermann MTA SZTAKI Defining and Running Parametric Study Workflow Applications by the P-GRADE Portal

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 2 Outline of the talk 1.Introduction to P-GRADE portal 2.PS-extension motivations 3.Principles of the PS-portal 4.Usage of the PS-portal 5.Distributing workflows among different Grid resources 6.Conclusions

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 3 Proxy management P-GRADE portal in a nutshell Definition of Grid resources Workflow creation Job mapping to Grid resources Workflow management and execution visualization

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 4 References P-GRADE Portal service is available for –Central European VO of EGEE –GILDA: Training VO of EGEE –HUNGRID VO of EGEE –SEE-GRID infrastructure –GIN VO –National Grids (UK National Grid Service, HunGrid, etc.) Under preparation for –US Open Science Grid, TeraGrid –Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE VOs, BioInfoGrid, BalticGrid –Many national Grids

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 5 User’s feedback Users want parameter study (PS) support at workflow level It means: –If the user has an existing workflow in a repository, he would like to run the same workflow (without any change) with many different parameters

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 6 Related research 1.PS at job level – main subject is scheduling 2.PS at workflow level – The workflow description supports the creation of PS runs 3.What our users want: executing workflows stored in workflow repositories with many inputs as PS

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 7 Principles of the PS-portal 1. Take the existing workflow as a black box 2. Modify only the external input ports to accept many input files 3. Create the cross-product of the input files and run the workflow with all combinations 4. Because of the large number of files both PS inputs and outputs are stored in the Grid! Storage elements 4 input files 3 input files 12 output files

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 8 Generating cross products and executable workflows (e-WFs) PS port: 4 instances of the input file PS port: 3 instances of the input file = 4 x 3 normal workflow execution 1 PS workflow execution

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 9 Steps of creating a PS-workflow 1. Start from a tested Workflow

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies Define the PS Port(s) Any external input port can be defined as PS- port

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 11 3.Define Subdirectory of Inputs PS port is represented by a directory and not by a single file

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies Define target Grid and subdirectory for the results

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies PS Workflow ready to Submit

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 14 Progress of submissions in PS detailed view Total = Init + Submitted + Rescue + Error + Finished

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 15 How to distribute workflows among different Grid resources? PS port: 4 instances of the input file PS port: 3 instances of the input file = Goal: Assign the 24 jobs to 24 different Grid resources 1 PS workflow execution

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 16 Single-Grid, Static Assignment PS port: 4 instances of the input file PS port: 3 instances of the input file = Assigns the 24 jobs to 2 Grid Resources within 1 Grid 1 PS workflow execution Assign resource A in Grid X Assign resource B in Grid X A A AA A A AA A A A A B B BB B B BB B B BB

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 17 Single-Grid, Dynamic Assignment via Broker PS port: 4 instances of the input file PS port: 3 instances of the input file = 1 PS workflow execution Assign to broker of Grid X Assign to broker of Grid X A B CD I J KL R S T V E F GH M N OP U X YZ Assigns the 24 jobs to 24 Grid Resources within 1 Grid

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 18 P-GRADE-Portal London Rome Athens TeraGrid (GT2/GT4) SEE-Grid (glite) P-GRADE portal is a multi-Grid portal

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 19 Static Multi-Grid Assignment PS port: 4 instances of the input file PS port: 3 instances of the input file = 1 PS workflow execution Assign to broker of Grid X Assign to broker of Grid Y XA XB XCXD XI XJ XKXL XR XS XT XV YE YF YGYH YM YN YOYP YU YX YYYZ Assigns the 24 jobs to 24 Grid Resources within 2 Grids

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 20 Dynamic Multi-Grid Assignment PS port: 4 instances of the input file PS port: 3 instances of the input file = 1 PS workflow execution Assign to meta-broker Assign to meta-broker XA XB YCXD YI XJ XKXL XR XS XT YV YE XF YGXH YM XN YOYP YU YX YYXZ Assigns the 24 jobs to 24 Grid Resources within 2 Grids

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 21 Conclusions Our solution: 1.Keeps both the workflow GUI and the parameter study support concept as simple as possible. 2.Enables run any existing workflow with different parameter sets without modifying the structure of the workflow. 3.Manages the execution of the workflows on as many Grid resources as possible. 4.Enables the collection of Grid resources from several Grids even if they are based on different Grid technology.

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 22 Thanks for your attention ? Further information:

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 23 Distributing workflows among different Grid resources 1.Single-Grid, Static Assignment: Statically assigning each node of the workflow to different resources in the same Grid (done) 2.Single-Grid, Dynamic Assignment: Dynamically assigning each node of the workflow to different resources in the same Grid using the broker of the Grid (done) 3.Static multi-Grid Assignment: Statically assigning different Grids to different nodes (done) 4.Dynamic multi-Grid Assignment: Dynamically assigning the nodes to different Grids using a meta-broker (under investigation)

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 24 Multi-Grid portals with meta-broker EGEE Grid UK NGS Broker Broker Meta- Broker Selects Grid for each node of the PS workflow Provides the largest achievable parallelism for PS workflows