EGEE-II INFSO-RI Enabling Grids for E-sciencE Grid application development with gLite and P-GRADE Portal Miklos Kozlovszky MTA SZTAKI

Enabling Grids for E-sciencE EGEE-II INFSO-RI Contents P-GRADE Portal in a nutshell Workflow development with the Portal Workflow execution with the Portal Scaling up to a parametric workflow

Enabling Grids for E-sciencE EGEE-II INFSO-RI Short History of P-GRADE portal Parallel Grid Application and Development Environment Initial development started in the Hungarian SuperComputing Grid project in 2003 It has been continuously developed since 2003 Detailed information: Open Source community development since January 2008:

Enabling Grids for E-sciencE EGEE-II INFSO-RI Download of OSS P-GRADE portal 110 downloads within the first month ~697 total downloads until now

Enabling Grids for E-sciencE EGEE-II INFSO-RI Main P-GRADE related projects EU SEE-GRID-1 ( ) –Integration with LCG-2 and gLite EU SEE-GRID-2,SEE-GRID-SCI ( / ) –Parameter sweep extension EU CoreGrid ( ) –To solve grid interoperation for job submission –To solve grid interoperation for data handling: SRB, OGSA-DAI GGF GIN (2006) –Providing the GIN Resource Testing portal EGEE 2,3 ( ) –Respect program tool used for training and application development ICEAGE ( ) –P-GRADE portal is used for training as official portal of the GILDA training infrastructure EU EDGeS ( ) –Transparent access to any EGEE and Desktop Grid systems

Enabling Grids for E-sciencE EGEE-II INFSO-RI References P-GRADE Portal service is available for –SEE-GRID-SCI infrastructure –Central European VO of EGEE –GILDA: Training VO of EGEE –Many national Grids (UK National Grid Service, GridIreland, Turkish Grid, Croatian Grid, etc.) –US Open Science Grid –Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE VOs, BioInfoGrid, BalticGrid –GIN VO of OGF –EGEE Respect program tool

Enabling Grids for E-sciencE EGEE-II INFSO-RI Portal installations

Enabling Grids for E-sciencE EGEE-II INFSO-RI Multi-Grid service portal To be used today!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Current situation and trends in Grid computing Fast evolution of Grid middleware: –GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, … Many production Grid systems are built with them –EGEE (LCG-2  gLite), UK NGS (GT2), Open Science Grid (GT2  GT4), NorduGrid (~GT2) Although the same set of core services are available everywhere, they are implemented in different ways –Data services –Computation services –Security services (single sign-on) –(Brokers)

Enabling Grids for E-sciencE EGEE-II INFSO-RI E-scientists’ concerns The P-GRADE Grid Portal gives you the answers! How to concentrate on my own research if the middleware I would like to use is in continuous change? How can I learn and understand the usage of the Grid? How can I develop Grid applications? How can I execute grid applications? How to tackle performance issues? How to use several Grids at the same time? How to migrate my application from one grid to another? How can I collaborate with fellow researchers?

Enabling Grids for E-sciencE EGEE-II INFSO-RI Motivations for developing P-GRADE portal P-GRADE portal should –Hide the complexity of the underlying grid middlewares –Provide a high-level graphical user interface that is easy-to-use for e-scientists –Support many different grid programming approaches:  Simple Scripts & Control (sequential and MPI job execution)  Scientific Application Plug-ins  Complex Workflows  Parameter sweep applications: both on job and workflow level  Interoperability: transparent access to grids based on different middleware technology (both computing and data resources) –Support several levels of parallelism

Enabling Grids for E-sciencE EGEE-II INFSO-RI Layers in a Grid system Basic Grid services: AA, job submission, info, … Higher-level grid services (brokering,…) Application toolkits, standards Application Grid middleware Command line tools P-GRADE Portal services Graphical interface

Enabling Grids for E-sciencE EGEE-II INFSO-RI Design principles of P-GRADE portal P-GRADE Portal is not only a user interface, it is a –General purpose –Workflow-level –Multi-Grid –Application Development and Execution Environment P-GRADE Portal includes a high-level middleware layer for orchestrating grid resources –inside a grid –among several different grids P-GRADE Portal is grid-neutral: –Unlike many existing grid portals it is not tailored to any particular grid type –Can be connected to various grids based on different grid middleware  LCG-2, gLite, GT2, GT4, ARC, Unicore, etc. –Implements the high-level grid middleware services on top of the existing grid middleware services –The workflow interface is the same no matter which type of grid is connected to it

Enabling Grids for E-sciencE EGEE-II INFSO-RI What is a P-GRADE Portal workflow? a directed acyclic graph where –Nodes represent jobs (batch programs to be executed on a computing element) –Ports represent input/output files the jobs expect/produce –Arcs represent file transfer operations semantics of the workflow: –A job can be executed if all of its input files are available

Enabling Grids for E-sciencE EGEE-II INFSO-RI Three Levels of parallelism – PS workflow level: Parameter study execution of the workflow – Workflow level: Parallel execution among workflow nodes (WF branch parallelism) Multiple jobs can run parallel Each job can be a parallel program – Job level: Parallel execution inside a workflow node (MPI job as workflow component) Multiple instances of the same workflow can process different data files

Enabling Grids for E-sciencE EGEE-II INFSO-RI times Example 1.: Computational Chemistry Department of Chemistry, University of Perugia SOLUTION OF SCHRODINGER EQUATION FOR TRIATOMIC SYSTEMS USING TIME- DEPENDENT (RWAVEPR) OR TIME INDEPENDENT (ABC) METHOD A single execution can be between 5 hours and 10 hours SEQUENTIAL FORTRAN 90 Many simulations at the same time

Enabling Grids for E-sciencE EGEE-II INFSO-RI x 10 x 25 x 5 x Hungarian Meteorology Service Forecasting dangerous weather situations (storms, fog, etc.), crucial task in the protection of life and property Processed information: surface level measurements, high-altitude measurements, radar, satellite, lightning, results of previous computed models Requirements: Execution time < 10 min High resolution (1km) Example 2.:Ultra-short range weather forecast

Enabling Grids for E-sciencE EGEE-II INFSO-RI Grid interoperation by P-GRADE Acccessing Globus, gLite and ARC based grids simultaneously P-GRADE portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Typical user scenario Compilation phase Portal server Grid services Certificate servers DOWNLOAD BINARI(ES) UPLOAD SOURCE(S) COMPILE – EDIT

Enabling Grids for E-sciencE EGEE-II INFSO-RI Typical user scenario Application development phase Portal server Grid services START EDITOR OPEN & EDIT or DEVELOP WORKFLOW SAVE WORKFLOW Certificate servers

Enabling Grids for E-sciencE EGEE-II INFSO-RI Certificate servers Portal server Grid services TRANSFER FILES, SUBMIT JOBS DOWNLOAD (SMALL) RESULTS Typical user scenario Workflow Execution phase VISUALIZE JOBS and WORKFLOW PROGRESS MONITOR JOBS DOWNLOAD PROXY CERTIFICATES

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE Portal structural overview User interface layer Presents the user interface Internal layer – Java classes Represents the internal concepts Java Webstart workflow editor Web browser EGEE and Globus Grid services (gLite WMS, LFC,…; Globus GRAM, GridFTP, …) Client P-GRADE Portal server Grid Grid layer – gLite and Globus command line tools Interfacing with grid services

Enabling Grids for E-sciencE EGEE-II INFSO-RI Interface layer User interface layer Java Webstart workflow editor Web browser Client Web server P-GRADE Portal server Gridpshere Web portal framework Gridsphere portlets P-GRADE portlets Workflow monitor: Java applet generator Workflow editor: Java webstart application

Enabling Grids for E-sciencE EGEE-II INFSO-RI Interface layer functionalities User interface layer Java Webstart workflow editor Web browser Client Web server P-GRADE Portal server Gridpshere Web portal framework Gridsphere portlets P-GRADE portlets Workflow monitor: Java applet generator Workflow editor: Java webstart application Workflow portlet Workflow manager, Storage, Upload Certificate portlet Upload, download and other operations Settings portlet Grid settings, Quota settings File management Manage files in the grid Compiler portlet Compile jobs on portal server Login Welcome...

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE vs. Non-P-GRADE portlets P-GRADE Portal portlets GridSphere 2.x Grid Portal framework

Enabling Grids for E-sciencE EGEE-II INFSO-RI Interface layer User interface layer Java Webstart workflow editor Web browser Client Web server P-GRADE Portal server Gridpshere Web portal framework Gridsphere portlets P-GRADE portlets Workflow monitor: Java applet generator Workflow editor: Java webstart application

Enabling Grids for E-sciencE EGEE-II INFSO-RI Interface layer User interface layer Java Webstart workflow editor Web browser Client Web server P-GRADE Portal server Gridpshere Web portal framework Gridsphere portlets P-GRADE portlets Workflow monitor: Java applet generator Workflow editor: Java webstart application

Enabling Grids for E-sciencE EGEE-II INFSO-RI Internal layer P-GRADE Portal server Grid layer Interfacing with grid services Gridsphere portletsP-GRADE portlets Workflow monitor: Java applet generator Workflow editor: Java webstart application “Tracefile” Java package Parses workflow monitoring information Workflow editor server (Java servlet) Workflow retrieval, upload Workflow state publication “Szupergrid” Java package: Workflow representation Resources configuration Quota management Certificate management Java interfaces

Enabling Grids for E-sciencE EGEE-II INFSO-RI Grid layer P-GRADE Portal server Grid layer Gridsphere portletsP-GRADE portlets Workflow monitor: Java applet generator Workflow editor: Java webstart application “Szupergrid” Java package: “Tracefile” Java package Workflow editor server (Java servlet) Workflow manager (Condor DAGMan) shell scripts Grid middleware clients: gLite User Interface Globus client packages EGEE and Globus Grid services (gLite WMS, LFC,…; Globus GRAM, …)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Grid layer P-GRADE Portal server Grid layer Gridsphere portletsP-GRADE portlets Workflow monitor: Java applet generator Workflow editor: Java webstart application “Szupergrid” Java package: “Tracefile” Java package Workflow editor server (Java servlet) Workflow manager (Condor DAGMan) shell scripts Grid middleware clients: gLite User Interface Globus client packages EGEE and Globus Grid services (gLite WMS, LFC,…; Globus GRAM, …) Client side command line tools and programming APIs to interact with gLite and Globus Grid Services

Enabling Grids for E-sciencE EGEE-II INFSO-RI Portlets/functionalities of P-GRADE portal Settings (portlet) Certificate and proxy management (portlet) Information system visualization (portlet) Graphical workflow editing Workflow manager (portlet) LFC (EGEE) file management (portlet) Compilation support (portlet) Fault-tolerance support

Enabling Grids for E-sciencE EGEE-II INFSO-RI Settings Portlet Portal administrator can –connect the portal to several grids –register the basic resources of the connected grids

Enabling Grids for E-sciencE EGEE-II INFSO-RI Settings Portlet User can customize the connected grids by adding and removing resources

Enabling Grids for E-sciencE EGEE-II INFSO-RI Certificate and proxy management Portlet User can upload his certificates of various grids to the MyProxy server User can download proxys and allocate to grids User can use simultaneously as many proxys as many grids are connected to the portal As a result parallel branches of a workflow can be executed simultaneously in several grids SEE-GRID access HUNGRID access

Enabling Grids for E-sciencE EGEE-II INFSO-RI MyProxy interaction in P-GRADE: Certificate Manager To start your session on the Grid you must create a proxy certificate on the portal server “Certificates” portlet: to upload a proxy into MyProxy servers to download a proxy from MyProxy into the portal server Certificates portlet

Enabling Grids for E-sciencE EGEE-II INFSO-RI Certificate Manager Downloading a proxy 1.MyProxy server access details: Hostname Port number User name (from upload) Password (from upload) 2.Proxy parameters: Lifetime Comment 3.Grid association

Enabling Grids for E-sciencE EGEE-II INFSO-RI Certificate Manager Associating the proxy with a grid This operation displays the details of the certificate and the list of available Grids (defined by portal administrator)

Enabling Grids for E-sciencE EGEE-II INFSO-RI EGEE Grid UK NGS P-GRADE-Portal London Paris Athens Solving Grid interoperation by P-GRADE Portal Different jobs can be parallel executed in different grids

Enabling Grids for E-sciencE EGEE-II INFSO-RI Interoperation vs. Interoperability Interoperation: –short term solution that defines what needs to be done to achieve interoperation between current production grids using existing technologies Interoperability: –native ability of Grids and Grid middleware to interact directly via common open standards As defined by the GIN (Grid Interoperation Now) CG (Community Group) of the OGF (Open Grid Forum) Grid 1Grid 2Grid 3 P-GRADE Portal Grid 1 Grid 2Grid 3 Interoperation Interoperability

Enabling Grids for E-sciencE EGEE-II INFSO-RI Information system Portlet

Enabling Grids for E-sciencE EGEE-II INFSO-RI Graphical workflow editing The aim is to define a DAG of batch jobs: 1.Drag & drop components: jobs and ports 2.Define their properties 3.Connect ports by channels (no cycles, no loops, no conditions) 4.Automatically generates JDL file

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Editor Properties of a job Properties of a job: Binary executable Type of executable Number of required processors Command line parameters The resource to be used for the execution: Grid/VO (Computing element)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Direct resource selection: Which computing element to use? The information system portlet queries BDII and GIIS servers I still don’t know which resource to use!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Automatic resource selection 1.Select a broker Grid/VO for the job (e.g. GILDA_LCG2_broker/GILDA_gLite_broker) 2.(Describe the ranks & requirements of the job in JDL) 3.The portal will use the broker to find the best resource for the job!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Editor Defining broker jobs Select a Grid with broker! (*_BROKER) Ignore the resource field! If default JDL is not sufficient use the built-in JDL editor!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Editor Built-in JDL editor JDL  look at the gLite Users’ manual!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Editor Defining input-output files File properties Type: input: the job reads output: the job generates File type: local: comes from my desktop remote: comes from an SE File: location of the file Internal file name: Executable reads the file in this name – fopen(“file.in”, …) File storage type (output files only): Permanent: final result Volatile: only data channel

Enabling Grids for E-sciencE EGEE-II INFSO-RI Client side location: result.dat LFC logical file name (LFC file catalog is required – EGEE VOs) lfn:/grid/gilda/kozlovszky/11-04_-_result.dat GridFTP address (in Globus Grids): gsiftp://somengshost.ac.uk/mydir/result.dat Local file Remote file How to refer to an I/O file? Client side location: c:\experiments\11-04.dat LFC logical file name (LFC file catalog is required – EGEE VOs) lfn:/grid/gilda/kozlovszky/11-04.dat GridFTP address (in Globus Grids): gsiftp://somengshost.ac.uk/mydir/11-04.dat Input fileOutput file

Enabling Grids for E-sciencE EGEE-II INFSO-RI Grid services Local vs. remote files Computing elements Storage elements REMOTE INPUT FILES REMOTE OUTPUT FILES LOCAL INPUT FILES & EXECUTABLES LOCAL OUTPUT FILES LOCAL INPUT FILES & EXECUTABLES LOCAL OUTPUT FILES Only the permanent files! Your binary can access data services directly too GridFTP API GFAL API lfc-*, lcg-* commands Portal server

Enabling Grids for E-sciencE EGEE-II INFSO-RI Job level file transfer in EGEE VOs Portal server EGEE VO Computing Element Storage Elements MyJob REMOTE INPUT FILE 23 gLite File Catalog Service REMOTE OUTPUT FILE REGISTER REMOTE OUTPUT FILE Custom file transfer Your code CAN but DOES NOT HAVE TO speak grid protocols EGEE WMS Pre script Binary of MyJob Post script 01 LOCAL FILES: Local input User binary Pre script Post script Generated by the portal Input sandbox LOCAL OUTPUT FILE Output sandbox Shell scripts

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow manager Lists available workflows Enables –Submitting –Aborting –Deleting existing workflows Shows status, logs and results of workflow executions Orchestrates job executions inside a workflow

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Management (workflow portlet) The portlet presents the status, size and output of the available workflow in the “Workflow” list It has a Quota manager to control the users’ storage space on the server The portlet also contains the “Abort”, “Attach”, “Details”, “Delete” and “Delete all” buttons to handle execution of workflows The “Attach” button opens the workflow in the Workflow Editor The “Details” button gives an overview about the jobs of the workflow

Enabling Grids for E-sciencE EGEE-II INFSO-RI White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

Enabling Grids for E-sciencE EGEE-II INFSO-RI White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

Enabling Grids for E-sciencE EGEE-II INFSO-RI White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

Enabling Grids for E-sciencE EGEE-II INFSO-RI White/Red/Green color means the job is initial/running/finished state Workflow Execution (observation by the workflow portlet)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Execution (observation by the workflow portlet) White/Red/Green color means the job is initialised/running/finished

Enabling Grids for E-sciencE EGEE-II INFSO-RI LFC (EGEE) file management

Enabling Grids for E-sciencE EGEE-II INFSO-RI Compilation support

Enabling Grids for E-sciencE EGEE-II INFSO-RI Logs provided for each job

Enabling Grids for E-sciencE EGEE-II INFSO-RI Analysis of the log :32:19 - Proxy with VOMS extensions created for VO "voce" with accounting group "" :32:19 - Job submission in progress :32:23 - Job has been submitted successfully! :32:23 - Job identifier is: " :32:26 - EGEE job's status has changed to "Waiting" (host is ) :33:00 - EGEE job's status has changed to "Ready" (host is ce1- egee.srce.hr) :35:46 - EGEE job's status has changed to "Waiting" (host is egee- ce.grid.niif.hu) :36:19 - EGEE job's status has changed to "Ready" (host is ce.cyf- kr.edu.pl) :36:53 - EGEE job's status has changed to "Waiting" (host is ce.cyf-kr.edu.pl) :37:26 - EGEE job's status has changed to "Done" (host is egee-ce.grid.niif.hu) :37:26 - Job found to be finished. Checking again if this is really the case :38:03 - EGEE job's status has changed to "Ready" (host is egee-ce1.gup.uni-linz.ac.at).

Enabling Grids for E-sciencE EGEE-II INFSO-RI Fault-tolerant Grid applications Utilizing –Condor DAGMan’s rescue mechanism –EGEE job resubmission mechanism of WMS If the EGEE broker leaves a job stuck in a CEs’ queue, the portal automatically –kills the job on this site and –resubmits the job to the broker by prohibiting this site. As a result –the portal guarantees the correct submission of a job as long as there exists at least one matching resource –job submission is reliable even in an unreliable grid

Enabling Grids for E-sciencE EGEE-II INFSO-RI Fault-tolerance by P-GRADE portal 09:33: the broker assigned the job to a site: ce1-egee.srce.hr 09:35: The broker moved the job to another site: egee-ce.grid.niif.hu 09:36: Again the broker moved the job to another site: ce.cyf-kr.edu.pl 09:37: The broker indicated that the job is Done, but. 09:38:... It turned out that the job was not finished (Done - Failed status), only it was moved to another site: egee-ce1.gup.uni-linz.ac.at 09:39: Again the broker moved the job to another site: ares02.cyf- kr.edu.pl 09:39: Again the broker moved the job to another site: ce.cyf-kr.edu.pl 09:40: After trying 10 different sites the VOCE broker gave it up and aborted the job (the Shallow RetryCount was set for 10): :40:16 - The job has been aborted!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Fault-tolerance by P-GRADE portal Our fault-tolerant portal did not give it up: :40:16 - The job can be submitted again (try 1 out of 3, excluding host(s): ce.cyf-kr.edu.pl) :40:17 - Proxy with VOMS extensions created for VO "voce" with accounting group "" :40:17 - Job submission in progress :40:27 - Job has been submitted successfully! :40:27 - Job identifier is: " :40:30 - EGEE job's status has changed to "Waiting" (host is ) :41:04 - EGEE job's status has changed to "Ready" (host is eszakigrid66.inf.elte.hu).

Enabling Grids for E-sciencE EGEE-II INFSO-RI :41:37 - EGEE job's status has changed to "Scheduled" (host is eszakigrid66.inf.elte.hu) :44:57 - EGEE job's status has changed to "Done" (host is eszakigrid66.inf.elte.hu) :44:57 - Job found to be finished. Checking again if this is really the case :45:34 - EGEE job's status has changed to "Waiting" (host is eszakigrid66.inf.elte.hu) :06:06 - The job's status hasn't changed for 20 minutes, resubmitting... It is a quite frequently occurring problem in EGEE-like grids that the broker leaves jobs stuck in CEs. queues.) In such case the portal automatically kills the job on this site and resubmits it to the broker :06:06 - Proxy with VOMS extensions created for VO "voce" with accounting group "" :06:06 - Job submission in progress :06:12 - Job has been submitted successfully! 10:10: The job successfully finished with exit code 0 on site: ce.ui.savba.sk Fault-tolerance by P-GRADE portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Lessons learnt P-GRADE portal provides –Easy-to-use but powerful workflow system (graphical editor, wf manager, etc.) –Three levels of parallelism  MPI job level  Workflow branch level  Parameter sweep at workflow level –Multi-grid/multi-VO access mechanism for various grids (LCG-2, gLite and GT2)  Simultaneous access  Transparent access  Migrating a workflow from one grid to another requires no modification in the workflow

EGEE-II INFSO-RI Enabling Grids for E-sciencE Thank you! Learn once, use everywhere Develop once, execute anywhere