1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences Grid Application Support by the P-GRADE Portal Gergely Sipos

2 CEOS WGISS Meeting, May 8-12, 2006, Budapest Outline Motivation of our work P-GRADE Portal in a nutshell Grid application development Grid application execution and monitoring Support for collaborative work Conclusions

3 CEOS WGISS Meeting, May 8-12, 2006, Budapest Layered view of Grid systems Basic Grid services: security, job submission, info, … Higher-level grid services: brokering, data replica, … Application toolkits, portals Applications Command line grid tools Graphical grid tools: P-GRADE Portal Resources to be shared Grid middleware: e.g. Globus, EGEE, … E-Scientists

4 CEOS WGISS Meeting, May 8-12, 2006, Budapest Motivation to the P-GRADE Portal Fast evolution of Grid middleware technologies and tools: –GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, … Many production Grids are now freely available for e- Scientists –EGEE (LCG-2 gLite), UK NGS (GT2), US Open Science Grid (GT2 GT4), NorduGrid (ARC), … The same set of services are available everywhere, but implemented in different ways –Computation services, data services, security services, (brokers) Lets provide a technology-neutral, intuitive graphical interface for the most common Grid middleware services!

5 CEOS WGISS Meeting, May 8-12, 2006, Budapest P-GRADE Portal in a nutshell General purpose, workflow-oriented computational Grid portal. Supports the development and execution of workflow-based Grid applications. Based on standard portlet framework (Gridsphere) –Easy to expand with new portlets (e.g. application-specific portlets) –Easy to tailor to community needs Grid services supported by the portal: ServiceEGEE gridsGlobus grids Job executionComputing ElementGRAM File storageStorage ElementGridFTP server Certificate managementMyProxy Information systemBDIIMDS-2 BrokeringWorkload Management System--- Job monitoringMercury Workflow & job visualizationPROVE The P-GRADE Portal hides middleware technologies and solves Grid interoperability problem at the workflow level

6 CEOS WGISS Meeting, May 8-12, 2006, Budapest What is a P-GRADE Portal workflow? a directed acyclic graph where –Nodes represent jobs (batch programs to be executed on a computing resource) –Ports represent input/output files the jobs expect/produce –Arcs represent file transfer operations semantics of the workflow: –A job can be executed if all of its input files are available

7 CEOS WGISS Meeting, May 8-12, 2006, Budapest Two levels of parallelism The workflow concept of the P-GRADE Portal enables the efficient parallelization of complex problems Semantics of the workflow enables two levels of parallelism: Each job can be a parallel program – Parallel execution inside a workflow node – Parallel execution among workflow nodes Multiple jobs run parallel

8 CEOS WGISS Meeting, May 8-12, 2006, Budapest 25 x 10 x 25 x 5 x Forecasting dangerous weather situations (storms, fog, etc.), crucial task in the protection of life and property Processed information: surface level measurements, high- altitude measurements, radar, satellite, lightning, results of previous computed models Requirements: Execution time < 10 min High resolution (1km) Ultra-short range weather forecast (Hungarian Meteorology Service)

9 CEOS WGISS Meeting, May 8-12, 2006, Budapest Multi-Grid portal: EGEE User Forum Job Repository (GEMLCA) Manchester Portal Client in Geneva Leeds P-GRADE Portal server Portal Server in London Executable NGS GT2 (UK) WestFocus GT4 Grid (UK) UoW Brunel Poznan Budapest EGEE (CE) Executable Service Invocation Executable

10 CEOS WGISS Meeting, May 8-12, 2006, Budapest The typical user scenario Part 1 - development phase Certificate servers Portal server Grid services START EDITOR OPEN & EDIT or DEVELOP WORKFLOW SAVE WORKFLOW

11 CEOS WGISS Meeting, May 8-12, 2006, Budapest Certificate servers Portal server Grid services TRANSFER FILES, SUBMIT JOBS DOWNLOAD RESULTS The typical user scenario Part 2 - execution phase VISUALIZE JOBS and WORKFLOW PROGRESS MONITOR JOBS DOWNLOAD PROXY CERTIFICATES SUBMIT WORKFLOW

12 CEOS WGISS Meeting, May 8-12, 2006, Budapest Opening the workflow editor The editor is a Java Webstart application download and installation is only a click!

13 CEOS WGISS Meeting, May 8-12, 2006, Budapest Workflow Editor Define DAG of batch jobs: 1.Drag & drop components: jobs and ports 2.Connect ports by channels (no cycles, no loops) 3.Define job and port properties

14 CEOS WGISS Meeting, May 8-12, 2006, Budapest Job property window Properties of a job: Binary executable (Type of executable) (Number of processors) Command line parameters The resource to be used for the execution: Grid Broker / resource

15 CEOS WGISS Meeting, May 8-12, 2006, Budapest Support for manual resource selection: information system browser The information system portlet queries EGEE and Globus information systems

16 CEOS WGISS Meeting, May 8-12, 2006, Budapest Support for broker-based resource selection 1.Select a broker Grid/VO for the job 2.(Specify extra ranks & requirements for the job in Job Description Language) 3.The broker will find the best resource for your job!

17 CEOS WGISS Meeting, May 8-12, 2006, Budapest Defining input/output data File type: input: required by the job output: produced by the job File location: local: my desktop remote: grid storage site File name: unique name of the file File storage type: Permanent: final result of the WF Volatile: only used for inter-job data transfer

18 CEOS WGISS Meeting, May 8-12, 2006, Budapest The typical user scenario Execution phase – step 1: Certificate servers Portal server Grid services DOWNLOAD PROXY CERTIFICATES

19 CEOS WGISS Meeting, May 8-12, 2006, Budapest Prerequisite of workflow execution: Download user certificates 1.Certificate server details: Hostname Port number User name Password 2.Certificate parameters: Lifetime

20 CEOS WGISS Meeting, May 8-12, 2006, Budapest Result of certificate download Multiple certificates for multiple Grids/VOs can be available on the portal server at the same time! SEE-GRID resources HUNGRID resources

21 CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

22 CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

23 CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

24 CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

25 CEOS WGISS Meeting, May 8-12, 2006, Budapest Workflow Execution (observation by the workflow portlet) White/Red/Green color means the job is initialised/running/finished

26 CEOS WGISS Meeting, May 8-12, 2006, Budapest On-Line application monitoring -The portal monitors and visualizes parallel jobs -The portal monitors and visualizes workflows: -Input data transfer -Job execution -Output data transfer

27 CEOS WGISS Meeting, May 8-12, 2006, Budapest Downloading the results… -Download small files in a single ZIP file -Keep large files in the Grid for future analysis!

28 CEOS WGISS Meeting, May 8-12, 2006, Budapest Collaborative work by the P-GRADE Portal Export and import workflows and execution results from/into the portal Share legacy applications as services with other portal users Share successfully executed jobs as services with other portal users Integrate other users legacy applications and jobs into your workflow

29 CEOS WGISS Meeting, May 8-12, 2006, Budapest P-GRADE Portal service is available for –SEE-GRID infrastructure –HUNGRID VO of EGEE –Central European VO of EGEE –GILDA: Training VO of EGEE –UK National Grid Service Under preparation for –US Open Science Grid, Economy-Grid, Swiss BioGrid, BioInfoGrid, Turkish Grid P-GRADE portal can be installed for any public/private EGEE or Globus Grid/VO Please contact us! References

30 CEOS WGISS Meeting, May 8-12, 2006, Budapest Main conclusion: Easy-to-use, technology-neutral Grid portal for e-Scientists The P-GRADE Portal hides differences of Grids Globus X – LCG2 – gLite Grid interoperability Transparent switching between Grid technologies Graphical tools for application development, execution and monitoring Seq & parallel components can be integrated into large Grid applications Your code does not have to contain grid specific calls Support for collaborative research Share workflows Share legacy applications and jobs as application services Built by standard portlet API customizable to specific application areas and user groups (such as earth scientists…)

31 CEOS WGISS Meeting, May 8-12, 2006, Budapest Further information… –Visit the site (manuals, slide shows, installation procedure, etc.) –Visit or request a training event! (see list of events on P-GRADE Portal homepage) Lectures, demos, hands-on tutorials, application development support –Get an account for a portal service: VOCE portal - SZTAKI SEEGRID portal – SZTAKI HUNGrid portal – SZTAKI NGS portal – University of Westminster –Install a portal for your community: If you are the administrator of a Grid/VO SZTAKI is pleased to help you install a portal for your community!

32 CEOS WGISS Meeting, May 8-12, 2006, Budapest P-GRADE Portal tutorial in Paris (free event!) 22-23, June – lectures, hands-on, application support See details on P-GRADE Portal homepage! Joint Regional CE EGEE and SEE-GRID Summer School on Grid Application Support 3-8 July, Budapest, SZTAKI Thank you! Questions?