Workflow level parametric study support by the P-GRADE portal Peter Kacsuk, Zoltan Farkas and Gabor Hermann MTA SZTAKI www.lpds.sztaki.hu

P-GRADE portal in a nutshell Proxy management Grid and Grid resource management Graphical editor for workflow creation Job mapping to Grid and Grid resources Workflow management and execution visualization PS Workflow P-GRADE portal

What is a P-GRADE Portal workflow? a directed acyclic graph where Nodes represent jobs (batch programs to be executed on a computing element) Ports represent input/output files the jobs expect/produce Arcs represent file transfer operations semantics of the workflow: A job can be executed if all of its input files are available PS Workflow P-GRADE portal

References P-GRADE Portal service is available for SEE-GRID infrastructure Central European VO of EGEE GILDA: Training VO of EGEE Many national Grids (UK National Grid Service, HunGrid, TurkGrid, CroGrid, etc.) US Open Science Grid, TeraGrid Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE VOs, BioInfoGrid, BalticGrid GIN VO PS Workflow P-GRADE portal

User’s feedback Users want parameter study (PS) support at workflow level It means: If the user has an existing workflow in a repository, he would like to run the same workflow (without any change) with many different parameters Goal of the work: Enable users to easily turn their workflow application into a parameter sweep application PS Workflow P-GRADE portal

Principles of the PS-portal 4 input files Storage elements Take the existing workflow as a black box Modify only the external input ports to accept several input files Create the cross-product of the input files and run the workflow with all combinations Because of the large number of files both PS inputs and outputs are stored in the Grid! Storage elements 3 input files 12 output files Storage elements PS Workflow P-GRADE portal

Parameter sweep (PS) workflow execution in P-GRADE portal 4 x 3 normal workflow execution PS port: 4 instances of the input file PS port: 3 instances of the input file = This provides the 3rd level of parallelism resulting a very large demand for Grid resources PS Workflow P-GRADE portal

Introducing three levels of parallelism Multiple instances of the same workflow with different data files Parallel execution among workflow nodes Multiple jobs run parallel Each job can be a parallel program Parallel execution inside a workflow node Parameter study execution of the workflow PS Workflow P-GRADE portal

PS_1_2 PS_1_1 PS DATA FLOW PS_2_2 PS_2_1 One or more Parameter Sets are defined. Each has different parameters. (Sets are represented by remote file directories,) PS_1_1 PS_2_1 PS_1_2 PS_2_1 Content of Parameter Sets are combined (Descartes product) PS_1_1 PS_2_2 Store the results of the calculations (Results are represented by remote files of one or more common directories) PS_1_2 PS_2_2 The workflow is executed as many times as many elements are in the Descartes product of input sets WF Result_1_1 Result_2_1 Result_1_2 Result_2_2 PS Workflow P-GRADE portal

Generator and Collector jobs To generate the necessary input file sets is a tedious work => portal support is needed Two kinds of Generator jobs: Auto-generator: Code is provided by the portal User gives only parameters Runs on the portal server User written generator: User can write any code Runs in the Grid Collector job: collects and processes the output files generated by the e-workflows Written by the user PS Workflow P-GRADE portal

An example Auto Generator PS Workflow P-GRADE portal

Data Model of PS Execution Input A_Gen Gen The Autogenerator is a special job running on the Portal server Coll Output Coll Output Coll Output PS Workflow P-GRADE portal

Three phases of PS execution First phase: executing ones all the Generators Second phase: executing all generated eWorkflows in parallel Last phase: executing ones all the Collectors PS Workflow P-GRADE portal

Monitoring PS-Workflow execution Collector phase becomes active when all eWorkflows have terminated PS Workflow P-GRADE portal

Multi-Grid Assignment and brokering 1 PS workflow execution Assigns the 24 jobs to 24 Resources within 2 Grids PS port: 4 instances of the input file XA XB XC XD YE YF YG YH PS port: 3 instances of the input file = XI XJ XK XL Assign to LCG-broker of Grid X (any EGEE VO) YM YN YO YP Assign to GTBroker of Grid Y (UK NGS, OSG, TeraGrid) XR XS XT XV YU YX YY YZ PS Workflow P-GRADE portal

Conclusions P-GRADE portal release 2.5: Keeps both the workflow GUI and the parameter study support concept as simple as possible. Enables run any existing workflow with different parameter sets without modifying the structure of the workflow. Extends the PS workflow with the concept of Generators and Collectors. Manages the execution of the workflows on as many Grid resources as possible. See the actual usage of the PS portal for solving an e-market place problem (EMMIL) at the SZTAKI demonstration booth PS Workflow P-GRADE portal

Thanks for your attention ? Further information: http://www.lpds.sztaki.hu/pgportal/ PS Workflow P-GRADE portal