1. Workflow level parametric study support by the P-GRADE portal
Peter Kacsuk, Zoltan Farkas and Gabor Hermann
MTA SZTAKI

2. P-GRADE portal in a nutshell
Proxy management
Grid and Grid resource management
Graphical editor for workflow creation
Job mapping to Grid and Grid resources
Workflow management and
execution visualization
PS Workflow P-GRADE portal

3. What is a P-GRADE Portal workflow?
a directed acyclic graph where
Nodes represent jobs (batch programs to be executed on a computing element)
Ports represent input/output files the jobs expect/produce
Arcs represent file transfer operations
semantics of the workflow:
A job can be executed if all of its input files are available
PS Workflow P-GRADE portal

4. References P-GRADE Portal service is available for
SEE-GRID infrastructure
Central European VO of EGEE
GILDA: Training VO of EGEE
Many national Grids (UK National Grid Service, HunGrid, TurkGrid, CroGrid, etc.)
US Open Science Grid, TeraGrid
Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE VOs, BioInfoGrid, BalticGrid
GIN VO
PS Workflow P-GRADE portal

5. User’s feedback
Users want parameter study (PS) support at workflow level
It means:
If the user has an existing workflow in a repository, he would like to run
the same workflow (without any change)
with many different parameters
Goal of the work:
Enable users to easily turn their workflow application into a parameter sweep application
PS Workflow P-GRADE portal

6. Principles of the PS-portal
4 input files
Storage elements
Take the existing workflow as a black box
Modify only the external input ports to accept several input files
Create the cross-product of the input files and run the workflow with all combinations
Because of the large number of files both PS inputs and outputs are stored in the Grid!
Storage elements
3 input files
12 output files
Storage elements
PS Workflow P-GRADE portal

7. Parameter sweep (PS) workflow execution in P-GRADE portal
4 x 3 normal workflow execution
PS port: 4 instances of the input file
PS port: 3 instances of the input file =
This provides the 3rd level of parallelism
resulting a very large demand for Grid
resources
PS Workflow P-GRADE portal

8. Introducing three levels of parallelism
Multiple instances of the same workflow with different data files
Parallel execution among workflow nodes
Multiple jobs run parallel
Each job can be a parallel program
Parallel execution inside a workflow node
Parameter study execution of the workflow
PS Workflow P-GRADE portal

9. PS_1_2
PS_1_1
PS DATA FLOW
PS_2_2
PS_2_1
One or more Parameter Sets are defined. Each has different parameters. (Sets are represented by remote file directories,)
PS_1_1
PS_2_1
PS_1_2
PS_2_1
Content of Parameter Sets are combined (Descartes product)
PS_1_1
PS_2_2
Store the results of the calculations
(Results are represented by remote files of one or more common directories)
PS_1_2
PS_2_2
The workflow is executed as many times as many elements are in the Descartes product of input sets WF
Result_1_1
Result_2_1
Result_1_2
Result_2_2
PS Workflow P-GRADE portal

10. Generator and Collector jobs
To generate the necessary input file sets is a tedious work => portal support is needed
Two kinds of Generator jobs:
Auto-generator:
Code is provided by the portal
User gives only parameters
Runs on the portal server
User written generator:
User can write any code
Runs in the Grid
Collector job:
collects and processes the output files generated by the e-workflows
Written by the user
PS Workflow P-GRADE portal

11. An example Auto Generator
PS Workflow P-GRADE portal

12. Data Model of PS Execution
Input
A_Gen
Gen
The Autogenerator is a special job running on the Portal server
Coll
Output
Coll
Output
Coll
Output
PS Workflow P-GRADE portal

13. Three phases of PS execution
First phase: executing ones all the Generators
Second phase: executing all generated eWorkflows in parallel
Last phase: executing ones all the Collectors
PS Workflow P-GRADE portal

14. Monitoring PS-Workflow execution
Collector phase becomes active when all eWorkflows have terminated
PS Workflow P-GRADE portal

15. Multi-Grid Assignment and brokering
1 PS workflow execution
Assigns the 24 jobs to 24
Resources within 2 Grids
PS port: 4 instances of the input file XA XB XC XD YE YF YG YH
PS port: 3 instances of the input file = XI XJ XK XL
Assign to LCG-broker of Grid X
(any EGEE VO) YM YN YO YP
Assign to GTBroker
of Grid Y (UK NGS, OSG, TeraGrid) XR XS XT XV YU YX YY YZ
PS Workflow P-GRADE portal

16. Conclusions P-GRADE portal release 2.5:
Keeps both the workflow GUI and the parameter study support concept as simple as possible.
Enables run any existing workflow with different parameter sets without modifying the structure of the workflow.
Extends the PS workflow with the concept of Generators and Collectors.
Manages the execution of the workflows on as many Grid resources as possible.
See the actual usage of the PS portal for solving an e-market place problem (EMMIL) at the SZTAKI demonstration booth
PS Workflow P-GRADE portal

17. Thanks for your attention?
Further information:
PS Workflow P-GRADE portal