1 Other features and next steps Gergely Sipos MTA SZTAKI Hungarian Academy of Sciences

2 Develop an application specific portlet: –To hide application logic from end users: Separate grid experts from grid users Complete P-GRADE development cycle Develop your jobs locally Compile your jobs for the grid –Compiler portlet Compose workflows from your jobs and execute it –Workflow editor, workflow manager, Information system portlets Scale up the workflow to a parameter study –Define input files (or add Input file generator) –Add output file collector

3 P-GRADE and user roles Portal Administrator: installs P-GRADE Portal 2.7 Grid Application Developer develops a grid application with P-GRADE Portal 2.7 sends the application to the grid portal developer End Users executes the published application with custom input parameters by creating application instances using the published application as a template Grid Portal Developer Creates new classes from the Application Specific Module for P-GRADE by changing the names of the classes develops a Gridsphere portlet that fits to the application I/O pattern and the end users’ needs connects the GUI to P-GRADE Portal using the programming API of P-GRADE Application Specific Module using the Application Specific Module he publishes the grid application and its GUI for end users

4 P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Portal Administrator installs the P-GRADE portal 2.7. Then adjusts the portlet to publish applications

5 P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Grid Application EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Application Developer creates own Grid Application.

6 P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Application specific portlet Grid Application EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Portal Developer develops a Application specific portlet using Java Server Pages and Gridsphere Tag Library Updates the the application specific package: connects Grid application to new portlet

7 P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Application specific portlet Grid Application Grid Application (Template) EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Portal Developer develops a Application specific portlet using Java Server Pages and Gridsphere Tag Library Updates the the application specific package: connects Grid application to new portlet

8 P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Application specific portlet Grid Application Grid Application (Template) Grid Application EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Application End users can create projects from a Published Grid Application Template. Then they can adjust parameters, and run in Grid via Application specific portlet!

9 I nterface to publish a P-GRADE Application Grid Application Developer can publish the application by a pre-developed portlet Button to revoke an already published application Button to publish the application Select an application

10 Matrix multiplication on EGEE end users’ point of view Do you see the grid somewhere?

11 Manage your computations Button to load a multiplication project Button to delete a multiplication project Button to create new multiplication project

12 Define a new multiplication task Name of the current project Matrix A (you can define values for Y) Matrix B is fixed

13 Define values for Y Parameters can be adjusted as Set, Range and Random.

14 Monitor progress Click to start computation Click to get status of computation Status bar of computation

15 Get result Click to Download Results

16 Part II Support of legacy codes, code reusability and grid interoperability

17 GEMLCA P-GRADE Portal P-GRADE Portal extended with GEMLCA Grid service back-end –To share jobs and legacy codes as application components with others –A step towards collaborative e-Science Developed by the University of Westminster (London) Support for Globus 4 grids (besides GT2 and EGEE) Available on the UK NGS and OGF GIN Portals P-GRADE Portal GEMLCA Globus 4 VOs Globus 2 VOs LCG / gLite VOs job Legacy code

18 Combine legacy codes with new codes inside the same workflow! Code invocation Job submission Combining legacy and non-legacy (job) components Publish successfully executed new code as legacy code

19 NGS P-GRADE Portal NGS P-GRADE portal is a production service of the UK NGS It has several advanced extensions to the original P-GRADE portal: –Integration with the GEMLCA legacy code service –Integration with SRB –Integration with OGSA-DAI The next part of the lecture shows these features of the NGS P-GRADE portal

20 Sharing jobs with other GEMLCA portal users New button for sucessfully executed jobs: create a legacy code service from this jobs

21 GEMLCA server Sharing jobs with other GEMLCA portal users Job Repository Put job into repository

22 Interoperation vs. Interoperability Interoperation: –short term solution that defines what needs to be done to achieve interoperation between current production grids using existing technologies Interoperability: –native ability of Grids and Grid middleware to interact directly via common open standards As defined by the GIN (Grid Interoperation Now) CG (Community Group) of the OGF (Open Grid Forum)

23 GIN areas of work ( middleware level) Information services: –identifying a subset of information items that can be used as a common minimum set –translation of these to and between GLUE and CIM (Common Information Model - used by NAREGI) Job submission –Job description: JSDL –JOB submission: Basic Execution Service interface (OGSA-BES) Data movement –interoperability between SRB and SRM data resources Authorization and identity management –developing policy and experimenting with cross-domain authorization using VOMS technology

24 Interoperation at workflow level Workflow level interoperation WF component executionData resources Intra-workflow Jobs or services of one particular workflow, Heterogeneous computing resources from different grids Jobs or services of one particular workflow, Heterogeneous data sources from different grids Inter-workflow Jobs or services of multiple workflow systems, Heterogeneous computing resources from different grids Jobs or services of multiple workflow systems, Heterogeneous data sources from different grids, Data conversion between WF systems

25 Grid interoperation 1 P-GRADE portal supports simultaneous job submission into several Grids: –GT2 (OSG, TeraGrid and UK NGS) –LCG-2, gLite – EGEE, SEEGRID –NorduGrid P-GRADE Portal enables: –Connecting 2 nd generation production Grids at workflow level

26 User P-GRADE Portal WPI Portal Server Connecting various production Grids gLite broker Certificate server Proxy Workflow ANL Fermi Lab OSG (GT2) SEEGRID (gLite) Job Budapest Istambul Athens Supports both direct and brokered job submission

27 Grid interoperation 2 P-GRADE Portal with GEMLCA supports legacy code service invocation in 2 nd and 3 rd generation Grids: –GT2, GT4 – (OSG, TeraGrid and UK NGS) –LCG-2, gLite – (EGEE, SEEGRID) Combination of jobs and legacy code services enables GEMLCA/P-GRADE Portal to support: –Extending 2 nd generation Grids with GT4 sites –Connecting 2 nd generation and 3 rd generation Grids

28 GEMLCA Repository Manchester Westminster User Oxford P-GRADE NGS GEMLCA Portal UoW Portal Server UoW site – GT4 with GEMLCA Service Service Invocation Executable Extending 2 nd generation Grids with 3 rd generation volunteer Grid sites UK NGS GT2

29 Connecting 2 nd generation and 3 rd generation production Grids Job Repository (GEMLCA) Fermi Lab User ANL GEMLCA P-GRADE Portal GIN Portal Server Executable OSG (GT2) WestFocus GT4 Grid UoW Brunel Poznan Budapest EGEE LCG/gLite Executable Service Invocation Executable

30 Interoperation at workflow level Workflow level interoperation WF component executionData resources Intra-workflow Jobs or services of one particular workflow, Heterogeneous computing resources from different grids Jobs or services of one particular workflow, Heterogeneous data sources from different grids Inter-workflow Jobs or services of multiple workflow systems, Heterogeneous computing resources from different grids Jobs or services of multiple workflow systems, Heterogeneous data sources from different grids, Data conversion between WF systems

31 Workflow Level Data Access Grid Interoperation Problem Different grids use different distributed file systems: –EGEE: SRM Grid FTP –UK NGS, OSG: Grid FTP SRB (SDSC (San Diego SuperComputing) Storage Resource Broker) Problem: How to run –a workflow where different nodes can be executed in different grids –and data files can also be taken from different grids?

32 J1 J2 J3 J4 J5 Grid 1 Grid 2 DB2 FS2 DB1 FS1 Workflow engine J: Job FS: File storage system, e.g. SRB or SRM DB: Database management system Intra-workflow interoperation of grid data resources Generic requirements

33 Workflow level integration of Grid data resources Types of integration Static: Data accessed statically before/after workflow execution Could be implemented as a standalone portlet: Portal 2.7 contains SRM portlet for EGEE NGS P-GRADE portal contains SRB and OGSA-DAI portlets Dynamic: Data request represented as an element of the workflow Port-level: data request is represented as either a data input or output of a node Fits well if data request is mainly data transfer (GridFTP, SRM, SRB) Node-level: data request is represented as a workflow node Fits well if data request is complex data transformation (e.g. ODSA-DAI)

34 4 portlets implemented 1.Data Service Manager 2.Database Browser 3.Query Manager 4.Data Manipulation Static OGSA-DAI integration a set of OGSA-DAI portlets –Implementation of a set of JSR 168 portlets –Support file delivery capabilities in order to transfer the data-sets before and after job execution

35 Extended PG-Portal DataBrowserWorkflow Step1: Select the OGSA-DAI service and DataBase resource with the DataBrowser OGSA-DAI service Oracle DataBase resource User eXist XML DataBase resource Request List of resources Request List of resources Storage resource (local or remote)

36 Step1: Select the OGSA-DAI service and DataBase resource with the DataBrowser Select service and resource

37 Extended PG-Portal DataBrowserWorkflow Step2: Run a query and deliver results to a (set of) file(s) OGSA-DAI service Oracle DataBase resource User eXist XML DataBase resource Query Results Query Results Query Storage resource (local or remote)

38 Step2: Run a query and deliver results to a (set of) file(s) Define query statement Deliver results to a set of files

39 Extended PG-Portal DataBrowserWorkflow Step3: Run (parameter study) workflow (this step does not use the OGSA-DAI portlets!) OGSA-DAI service Oracle DataBase resource User eXist XML DataBase resource Create workflow Workflow output Storage resource (local or remote) Input data

40 Extended PG-Portal DataBrowserWorkflow Step4: Transfer results back into an OGSA-DAI database OGSA-DAI service Oracle DataBase resource User eXist XML DataBase resource Workflow output Storage resource (local or remote) Workflow output Confirmation Request

41 Step4: Transfer results back into an OGSA-DAI database Define target database Define target table Define files to be transferred and converted

42 A P-GRADE job represents an OGSA-DAI query, update or workflow OGSA-DAI clients are deployed as GEMLCA legacy codes and placed into the GEMLCA repository –document client: executes general OGSA- DAI workflows represented as request documents –query client: submits query statements to a given resource –update client: submits query/update statements to a given resource. The user simply selects an appropriate OGSA-DAI client and set parameter values Dynamic node-level OGSA-DAI integration OGSA-DAI workflows are represented as P-GRADE jobs

43 Executes the given OGSA- DAI query Analysis on the sample data Selecting Grid Selecting OGSA-DAI client Setting OGSA-DAI service URL Setting Database Resource ID Generates sampler queries Gathering results OGSA-DAI workflow integration OGSA-DAI clients exposed with GEMLCA – a case study Setting query file: can be uploaded or produced by another job Query file Results in CSV file

44 Running at OSG Running at EGEE Running at NGS From NGS SRB (both) From NGS SRB From NGS GFTP From local (both) From NGS SRB To EGEE SRM Running at NGS From NGS GFTP To NGS SRB Workflow level Interoperation of local, SRB, SRM and GridFTP file systems Dynamic port-level integration of Grid data resources: Different port types, including local, remote and SRB ports, can be freely mixed as input or output of workflow components. This solution allows the seamless interoperation of SRB catalogues, GridFTP file systems and EGEE storage elements

45 So here we are! GridFTP, SRM, SRB and OGSA-DAI workflow level integration Portal server Grid infrastructure Computing resources GridFTP servers REMOTE INPUT FILES REMOTE OUTPUT FILES LOCAL INPUT FILES LOCAL OUTPUT FILES LOCAL INPUT FILES LOCAL OUTPUT FILES User level storage EGEE Storage elements Control of remote input/output Workflow level Interoperation of local, GridFTP, SRM and SRB file catalogues and databases exposed by OGSA-DAI SRB servers OGSA- DAI services Data manipulation Input to workflows Output from workflows

46 Lessons learnt P-GRADE portal became popular because it provides –Easy-to-use but powerful workflow system (graphical editor, wf manager, etc.) –Easy-to-use parameter sweep concept support –Easy-to-use MPI program execution support –Multi-grid/multi-VO access mechanism (job submission grid interoperation at workflow level) for LCG-2, gLite and GT2 Its extension with GEMLCA, SRB and OGSA-DAI enables –The usage of legacy codes as grid-enabled services –The usage of service/job repository –Access to SRB data resources and OGSA-DAI database services –Multi-grid/multi-VO access mechanism for LCG-2, gLite, GT2 and GT4 –Solving grid interoperation for grid data services

47 Lessons learnt If you use P-GRADE portal you are not connected to any grid Your application can seamlessly run on many grid systems where you have valid certificate As the grid technology progress you do not have to port your application for the new grid middlewares (it is the portal that should support the new grid middlewares). The sustainability of P-GRADE portal is guaranteed since –It is OSS –There is a Portal Developer Alliance that continuously improve the portal The P-GRADE Portal Developer Alliance is open for everyone to join. See details at:

48 Next steps How to get involved

49 Portal installations P-GRADE Portal services: –SEE-GRID infrastructure –Central European VO of EGEE –GILDA: Training VO of EGEE –Many national Grids (UK National Grid Service, HunGrid, Turkish Grid, etc.) –US Open Science Grid, TeraGrid –OGF Grid Interoperability Now (GIN) VO –… Portal services and account request: Account request form on portal login page

50 Installing P-GRADE Portal –(If you are the administrator of a Grid/VO then) get the installation package from SourceForge Server requirements Local user support must be provided by you –If you are member of an EGEE / Globus VO that does not have P-GRADE Portal yet, then Ask SZTAKI to add your VO to the Multi-grid P-GRADE Portal Installation Ask the manager of your VO to install P-GRADE Portal

51 To configure your VO in Multi-grid P-GRADE Portal Send the following information to P-GRADE Portal team: Technical –WMS URL –BDII URL –LFC host URL –Default SE of the VO –VOMS server URL, certificate subject –VOMS server certificate –Extra CA packages if VO members use other than certificates. Administrative –Homepage of VO/community –Use cases for P-GRADE –Number of users

52 Support services Training: –Training events, trainer community: list: –Portal to access GILDA resources –Online tutorials User forum Installation and operational support team Application Development and Porting Support:

53 Thank you! Learn once, use everywhere Develop once, execute anywhere