1. Introduction to the SHIWA Simulation Platform EGI User Forum, 21. 04
Introduction to the SHIWA Simulation Platform EGI User Forum, Vilnius Tamas Kiss – Gabor Terstyanszky – University of Westminster

2. SHIWA Simulation Platform at a glance
SHIWA Simulation Platform v1 (SSP v1):
deployed at production level since March 2011
supports coarse-grained workflow interoperability
native workflow system is the P-GRADE workflow system
enables either embedding or invoking ASKALON, GWES, Kepler, MOTEUR, Taverna and Triana workflows
allows e-scientists to search workflows uploaded to the SHIWA Workflow Repository and use them in their experiments

3. Overall architecture of SSP v1
SHIWA Science Gateway
SHIWA VO
SHIWA Portal
SHIWA Repository
local cluster
gLite DCI
Globus DCI
WF1
WFn
GEMLCA admin
Triana WE
WF1
WFm
Taverna WE
WE1
WEp
MOTEUR WE
ASKALON WE
PGRADE Workflow editor
GEMLCA Repository
Kepler WE
MOTEUR WE
GWES WE
PGRADE Workflow engine
ASKALON WE
pre-deployed- WEs
GEMLCA with GIB
GEMLCA Service

4. PGRADE Workflow editor PGRADE Workflow engine
SHIWA Science Gateway
SHIWA Portal:
P-GRADE 2.4 portal technology
certificate/proxy and DCI resource management
access to different DCI information systems
integrated with the P-GRADE Workflow System (used as native workflow engine)
administration of GEMLCA services
GEMLCA Service:
converts legacy applications such as workflows and workflow engines into Grid services
invokes locally or remotely pre-deployed workflow engines or submits workflow engines to local or remote resources to execute workflows
GEMLCA Repository
manage workflow engine (WE) and workflow (WF) data to support execution
SHIWA Repository:
create, add, edit and delete workflow metadata
upload and download workflows with their implementations and configurations.
SHIWA Science Gateway
SHIWA Portal
SHIWA Repository
WF1
WFn
GEMLCA admin
WF1
WFm
WE1
WEp
PGRADE Workflow editor
GEMLCA Repository
PGRADE Workflow engine
GEMLCA with GIB
GEMLCA Service

5. SHIWA VO VO services: Logical File Catalogue: Computing elements:
lfc-egee.in2p3.fr
Computing elements:
karwendel.dps.uibk.ac.at – Austrian NGI (Globus)
othello.zih.tu-dresden.de – German NGI (Globus)
ngs.wmin.ac.uk – UK NGI (Globus)
phoebe.deimos.htc.biggrid.nl – Dutch NGI (g-Lite)
Storage resources:
carme.htc.biggrid.nl (9.9TB)
Workflow engines:
Moteur engine to submit to g-Lite resources
Askalon and GWES engines to submit to Globus resources
Triana, Taverna, Moteur, Kepler and Askalon engines deployed locally at UoW
SHIWA VO
local cluster
gLite DCI
Globus DCI
Triana WE
Taverna WE
MOTEUR WE
ASKALON WE
Kepler WE
MOTEUR WE
GWES WE
ASKALON WE
pre-deployed- WEs

6. Family of P-GRADE Portal products
SHIWA Portal – based on P-GRADE portal
Family of P-GRADE Portal products
P-GRADE portal
Creating (basic) workflows and parameter sweeps for clusters, service grids, desktop grids
P-GRADE/GEMLCA portal (University of Westminster)
To wrap legacy applications into Grid Services
To add legacy code services to P-GRADE Portal workflows
WS-PGRADE
Creating complex workflow and parameter sweeps for clusters, service grids, desktop grids, databases
Creating complex applications using embedded workflows, legacy codes and community components from workflow repository

7. P-GRADE/GEMLCA Portal in a nutshell
General purpose, workflow-oriented computational Grid portal. Supports the development and execution of workflow-based Grid applications – a tool for Grid orchestration
Based on GridSphere-2
Easy to expand with new portlets (e.g. application-specific portlets)
Easy to tailor to end-user needs
Developed by SZTAKI and the P-GRADE portal alliance
Grid services supported by the portal:
Service
EGEE grids (LCG/gLite)
Globus grids
Job execution
Computing Element
GRAM
File storage
Storage Element
GridFTP server, SRB
Certificate management
MyProxy
Information system
BDII
MDS-2, MDS-4
Brokering
Workload Management System
GTbroker
Job monitoring
Mercury
Workflow & job visualization
PROVE
Legacy Code Management
GEMLCA

8. GEMLCA Repository Entities in the repository: Actors:
legacy code repository which stores workflows and workflow engines with their Legacy Code Interface Descriptions (LCID)
Entities in the repository:
workflows LCID + data (implementation + configuration)
workflow engines - LCID + data (binary + configuration)
Actors:
e-scientists: browse the repository and simply select the application to include in their workflows
workflow developers: create, upload and manage workflow’s LCID and data into the repository
workflow engine developers: create, upload and manage workflow engine’s LCID and data into the repository.

9. GEMLCA Repository ports are defined automatically
command line parameters are displayed automatically

10. SHIWA Workflow Repository
a workflow repository with extended metadata support
Entities in the repository:
workflows - describe the workflow behaviour
implementations - everything that is necessary for the workflow engine to be able to run the workflow
configurations - input files and parameters of workflows
Actors:
e-scientists: search/browse the repository to find and download workflows
workflow developers: enable e-scientists to run workflows - define workflows with their implementations and configurations, and upload and publish them
repository administrators: manage the repository as system administrators.

11. SHIWA Repository Data Structure
Workflow description
plain text describing inputs and outputs and explains what the workflow does
Workflow implementations
all implementation files or references to these files
data/metadata necessary to run the workflow on its associated workflow engine.
Workflow configurations
pre-defined input files and parameters or references to these, and other data/metadata of a workflow.
Workflow engines
files and other data/metadata necessary to execute a workflow engine on a grid site or references to them

12. SHIWA Workflow Repository

13. Workflow engine upload PGRADE Workflow editor PGRADE Workflow engine
SSP v1 – User scenarios
Workflow engine upload
SHIWA Science Gateway
SHIWA Portal
SHIWA Repository
Actor: WE developer
Frequency: once for a particular version of a WE
Process:
step 1 - create and upload workflow engine’s legacy code description (LCID) using the GEMLCA admin portlet.
step 2 - upload the workflow engine’s binary to the GEMLCA Repository (WE1, …WEp)
WF1
WFn s1
create LCID
GEMLCA admin
WE+ LCID
upload WE s2
WF1
WFm
WE developer
WE1
WEp
PGRADE Workflow editor
GEMLCA Repository
PGRADE Workflow engine
GEMLCA with GIB
GEMLCA Service

14. PGRADE Workflow editor PGRADE Workflow engine
SSP v1 – User scenarios
Workflow upload
SHIWA Science Gateway
Actor: WF developer
Frequency: once for a particular version of a WF
Process:
step 1 - create and upload the workflow’s LCID description using the GEMLCA admin portlet
step 2 - upload the workflow descriptor file to the GEMLCA Repository (WF1, …WFm).
step 3 - define the workflow metadata using the SHIWA Repository GUI
step 4 - upload the workflow descriptor file to SHIWA Repository (WF1, …WFn).
SHIWA Portal
SHIWA Repository
create WF metadata s3
WF1
WFn
upload WF s4 s1
create LCID
GEMLCA admin
WF + LCID
upload WF s2
WF developer
WF1
WFm
WE1
WEp
PGRADE Workflow editor
GEMLCA Repository
PGRADE Workflow engine
GEMLCA with GIB
GEMLCA Service

15. Workflow creation and execution – invoking pre-deployed WF engines
SSP v1 – User scenarios
Workflow creation and execution – invoking pre-deployed WF engines
SHIWA Repository
SHIWA Portal
WF1
e-scientist
SHIWA Science Gateway
invoke WE
GEMLCA Service
WFn
WE1
WEp
search WF
GEMLCA Repository
edit WF
WE + WF
WFm
GEMLCA with GIB s1 s2 s3 s5 s4
WF list
PGRADE Workflow engine
PGRADE Workflow editor
gLite DCI
MOTEUR WE
GWES WE
Globus DCI
pre-deployed- WEs
Kepler WE
Taverna WE
Triana WE
local cluster
ASKALON WE
SHIWA VO
GEMLCA admin

16. Workflow creation and execution – invoking pre-deployed WF engines
SSP v1 – User scenarios
Workflow creation and execution – invoking pre-deployed WF engines
Actor: e-scientist or workflow developer
Goal: to run the embedded or non-native workflows on locally or remotely pre-deployed workflow engines
step 1 - browse the SHIWA repository, select workflows to execute
step 2 - create embedded workflows by adding non-native workflows selected in step 1 using their workflow IDs
step 3 - P-GRADE workflow engine submits the embedded workflow through GEMLCA Service
step 4 - the GEMLCA Service combines the LCIDs of the workflow and its corresponding workflow engine.
step 5 - the GEMLCA Service invokes the pre-deployed workflow engine to execute the workflow on the target Distributed Computing Infrastructure

17. Workflow creation and execution – submitting WF engines to target DCI
SSP v1 – User scenarios
Workflow creation and execution – submitting WF engines to target DCI
SHIWA Repository
SHIWA Portal
WF1
e-scientist
GEMLCA admin
SHIWA Science Gateway
WFn
WE1
WEp
search WF
GEMLCA Repository
edit WF
WE + WF
WFm s1 s2 s3 s4
WF list
PGRADE Workflow engine
PGRADE Workflow editor
pre-deployed- WEs
submit WE s5
GEMLCA Service
GEMLCA with GIB
gLite DCI
MOTEUR WE
GWES WE
Globus DCI
Kepler WE
Taverna WE
Triana WE
local cluster
ASKALON WE
SHIWA VO

18. Workflow creation and execution – submitting WF engines to target DCI
SSP v1 – User scenarios
Workflow creation and execution – submitting WF engines to target DCI
Actor: e-scientist or workflow developer
Goal: to submit the workflow engines to local or remote resources to execute the embedded or non-native workflows
step 1 - browse the SHIWA repository, select workflows to execute
step 2 - create embedded workflows by adding non-native workflows selected in step 1 using their workflow IDs
step 3 - P-GRADE workflow engine submits the embedded workflow through GEMLCA Service
step 4 - the GEMLCA Service combines the LCIDs of the workflow and its corresponding workflow engine.
step 5 - the GEMLCA Service submits the workflow engine with the selected workflow to the target DCI (in this case local cluster)

19. SSP v1 – How to get access? Register with the SHIWA VO:
aim: to get authorisation to access GEMLCA resource, WF engines, compute and data resources
Get an account to the SHIWA portal
aim: to get access to the portal
Register with the SHIWA repository
aim: to be authorised to upload and modify workflows
repository can be browsed and workflows can be downloaded without account
Detailed how-to-get-access guide:

20. SHIWA Simulation Platform - the future
SHIWA Simulation Platform v2 (SSP v2):
to be operational by March 2012
supports both coarse- and fine-grained workflow interoperability
not only the P-GRADE but the ASKALON and MOTEUR workflow systems could also be native workflow systems
supports the SHIWA Desktop solution
converts workflows from one workflow representation to another using the IWIR Interpreter

21. Thank you. Questions? SHIWA Portal: SHIWA Workflow Repository
SHIWA Workflow Repository