1. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org P-GRADE overview and introduction: workflows & parameter sweeps (Advanced features) Miklos Kozlovszky MTA SZTAKI m.kozlovszky@sztaki.hu

2. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 2 Parameter Sweep applications Typical usage of the Grid is that the same application is executed with many different parameters P-GRADE portal from release 2.5 supports this functionality The next part of the lecture shows how parameter sweep (PS) applications are supported by P-GRADE portal

3. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 3 Scaling up a workflow to a parameter study with P-GRADE Complete workflow Output file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myoutputs) Input file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myinputs)

4. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 4 Turning a WF into a parameter study By turning at least one of the open input ports into a “PS Input port” the WF is turned into a Parameter Study WF

5. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 5 Input-output files are stored in SEs /grid/gilda/kozlovszky/InputImages Image.0 Image.1 /grid/gilda/kozlovszky/XCoordinates XCoordinate.0 XCoordinate.1 /grid/gilda/kozlovszky/YCoordinates YCoordinate.0 YCoordinate.1 /grid/gilda/kozlovszky/Output workflow1.zip workflow2.zip... Cross product of inputs: 2 x 2 x 2 = 8 execution of the whole workflow (8 e-workflows are generated

6. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 6 PS Input Port of Simple PS Remote file Directory instead of FILE reference Do not use the prefix lfn: if the directory is EGEE Grid file catalogue

7. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 7 Simple PS Activity 2: placement of result Properties of the VO File Catalog Menu item PS Properties is available in the Workflow menu The Output directory will contain the ZIP-ped result files of the Workflow executions One SE of the VO – Choose a reliable one!

8. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 8 Executing PS workflows PS Details for parameter sweep workflow applications

9. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 9 Workflow Manager List PS Details view showing eWF-s New, middle level list to render the details of a PS Workflow. (Status of complete application) Statistics shows the progress of the elaboration of the whole PS The eWorkflow buffer list shows the state of the Workflows being processed.

10. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 10 Details view of the eWF Ax_EQU_B_voce_PS.6 Job level details of an eWorkflow See, that the button Attach is missing as there is not to much importance to access the WE until the eWorkflow list is exhausted

11. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org Advanced PS WFs in P-GRADE Portal

12. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 12 Advanced parameter studies Generator component(s) Initial input data Generate or cut input into smaller pieces Collector component(s) Aggregate result Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myinputs ) Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myoutputs) Complete workflow

13. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 13 Concept of advanced parameter study workflows GEN SEQ COLL SEQ Parameter study part Collector part evaluates and integrates the results Generator part generates the input parameter space

14. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 14 The generic PS workflow application Third phase: executing the collectors Second phase: executing the “real” processor jobs for each input file combination First phase: executing the generators

15. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 15 Data Model of advanced PS Gen A_Gen Input Coll Output Coll Output Coll Output The Autogenerator is a special job: May have no input Runs on the Portal

16. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 16 Parameter generator Generator can be attached to any parameter input port Generator can be Auto generator: to generate text files Custom generator: to generate any content Generated files are moved into SE by the portal

17. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 17 Definition Window of Auto Generator Job User defines the template of the text file User puts keys into the template User defines values for the keys Integer number Real number Custom set … By clicking on a key the definition window for this key is opened.

18. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 18 (Auto) Generator Attribute Editor for SE definition Attribute Editor defines the properties of remote files created by the Generator: 1. Storage Element must be defined if an LCG like (EGEE ) file access has been defined in the PS Output Port belonging to the Generator

19. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 19 Detailed view of a PS workflow Workflow instances Overall statistics of workflow instances Collector job(s) Generator job(s)

20. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org PS practical

21. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 21 Matrix multiplication PS 10 PS workflows Multiplication job 1 2 3 4 5 6 7 8 12 2 1 3 1 1 1 3 3 3 Matrix2 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 15...

22. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 22 Exercise Matrix multiplication PS Multiplication job Auto generator Input files stored on SEs and registered in LFC catalog 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 12 1 2 3 4 5 6 7 8 15 1 2 3 4 5 6 7 8 Y 9 <= Y <=36, step 3 2 1 3 1 1 1 3 3 3 Matrix2 Output files stored on SEs and registered in LFC catalog...