EGEE-II INFSO-RI Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows & parameter sweeps (Advanced features) Miklos Kozlovszky MTA SZTAKI

Enabling Grids for E-sciencE EGEE-II INFSO-RI Parameter Sweep applications Typical usage of the Grid is that the same application is executed with many different parameters P-GRADE portal from release 2.5 supports this functionality The next part of the lecture shows how parameter sweep (PS) applications are supported by P-GRADE portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Scaling up a workflow to a parameter study with P-GRADE Complete workflow Output file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myoutputs) Input file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myinputs)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Turning a WF into a parameter study By turning at least one of the open input ports into a “PS Input port” the WF is turned into a Parameter Study WF

Enabling Grids for E-sciencE EGEE-II INFSO-RI Input-output files are stored in SEs /grid/gilda/kozlovszky/InputImages Image.0 Image.1 /grid/gilda/kozlovszky/XCoordinates XCoordinate.0 XCoordinate.1 /grid/gilda/kozlovszky/YCoordinates YCoordinate.0 YCoordinate.1 /grid/gilda/kozlovszky/Output workflow1.zip workflow2.zip... Cross product of inputs: 2 x 2 x 2 = 8 execution of the whole workflow (8 e-workflows are generated

Enabling Grids for E-sciencE EGEE-II INFSO-RI PS Input Port of Simple PS Remote file Directory instead of FILE reference Do not use the prefix lfn: if the directory is EGEE Grid file catalogue

Enabling Grids for E-sciencE EGEE-II INFSO-RI Simple PS Activity 2: placement of result Properties of the VO File Catalog Menu item PS Properties is available in the Workflow menu The Output directory will contain the ZIP-ped result files of the Workflow executions One SE of the VO – Choose a reliable one!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Executing PS workflows PS Details for parameter sweep workflow applications

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Manager List PS Details view showing eWF-s New, middle level list to render the details of a PS Workflow. (Status of complete application) Statistics shows the progress of the elaboration of the whole PS The eWorkflow buffer list shows the state of the Workflows being processed.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Details view of the eWF Ax_EQU_B_voce_PS.6 Job level details of an eWorkflow See, that the button Attach is missing as there is not to much importance to access the WE until the eWorkflow list is exhausted

EGEE-II INFSO-RI Enabling Grids for E-sciencE Advanced PS WFs in P-GRADE Portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Advanced parameter studies Generator component(s) Initial input data Generate or cut input into smaller pieces Collector component(s) Aggregate result Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myinputs ) Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myoutputs) Complete workflow

Enabling Grids for E-sciencE EGEE-II INFSO-RI Concept of advanced parameter study workflows GEN SEQ COLL SEQ Parameter study part Collector part evaluates and integrates the results Generator part generates the input parameter space

Enabling Grids for E-sciencE EGEE-II INFSO-RI The generic PS workflow application Third phase: executing the collectors Second phase: executing the “real” processor jobs for each input file combination First phase: executing the generators

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Model of advanced PS Gen A_Gen Input Coll Output Coll Output Coll Output The Autogenerator is a special job: May have no input Runs on the Portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Parameter generator Generator can be attached to any parameter input port Generator can be Auto generator: to generate text files Custom generator: to generate any content Generated files are moved into SE by the portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Definition Window of Auto Generator Job User defines the template of the text file User puts keys into the template User defines values for the keys Integer number Real number Custom set … By clicking on a key the definition window for this key is opened.

Enabling Grids for E-sciencE EGEE-II INFSO-RI (Auto) Generator Attribute Editor for SE definition Attribute Editor defines the properties of remote files created by the Generator: 1. Storage Element must be defined if an LCG like (EGEE ) file access has been defined in the PS Output Port belonging to the Generator

Enabling Grids for E-sciencE EGEE-II INFSO-RI Detailed view of a PS workflow Workflow instances Overall statistics of workflow instances Collector job(s) Generator job(s)

EGEE-II INFSO-RI Enabling Grids for E-sciencE PS practical

Enabling Grids for E-sciencE EGEE-II INFSO-RI Matrix multiplication PS 10 PS workflows Multiplication job Matrix

Enabling Grids for E-sciencE EGEE-II INFSO-RI Exercise Matrix multiplication PS Multiplication job Auto generator Input files stored on SEs and registered in LFC catalog Y 9 <= Y <=36, step Matrix2 Output files stored on SEs and registered in LFC catalog...