1. Introduction to gUSE and WS-PGRADE portal
Gergely Sipos
MTA SZTAKI

2. Outline History, family of P-GRADE products WS-PGRADE features
P-GRADE Portal, WS-PGRADE, gUSE
WS-PGRADE features
Scalable architecture
Seamless access to various types of resources
Comfort features
Separated views for end users and developers
Advanced data-flows
Users and applications
Summary and Next steps

3. Family of P-GRADE Portal products
Creating (basic) workflows and parameter sweeps for gLite and Globus middleware based utility grids
P-GRADE/GEMLCA portal (University of Westminster)
To wrap legacy applications into Grid Services
To add legacy code services to P-GRADE workflows
(No parameter sweep support!)
WS-PGRADE
Creating complex workflow and parameter sweeps for local clusters, utility grids and desktop grids
Creating complex applications using embedded workflows, legacy codes and community components from repository
Apply for an account on Beta release
Browse the User manual

4. P-GRADE Grid Portal Pros. Cons.
Easy-to-use workflow system with graphical editor
Easy-to-use parameter sweep concept at workflow level
Multi-grid / multi-VO access mechanism: job submission to LCG (~old gLite), gLite and Globus Toolkit 2
Intelligent handling of grid errors
Open source community on Sourceforge
Reliable, production installations for several Grid, EGEE VOs
Part of EGEE RESPECT programme
Cons.
Considered too simple for some IT end users, while too complecated for some non-IT end users
Workflow features found limited for some applications
Internal structure is monolithic  hard to build developer community around it

5. Motivations of creating gUSE
To overcome (most of) the limitations of P-GRADE Portal:
To provide better modularity
To improve scalability
To enable advanced dataflow patterns
To interface with wider range of resources
To separate Application Developer view from Application End User view
New products: WS-PGRADE (Web Services Parallel Grid Runtime and Developer Environment) and gUSE (Grid User Support Environment) architecture

6. WS-PGRADE – gUSE architecture
Graphical User Interface: WS-PGRADE
Gridsphere portlets
gUSE
Workflow storage
File storage
File storage
gUSE information system
Autonomous Services: high level middleware service layer
Workflow Engine
Application repository
Submitters
Submitters
Submitters
Meta-broker
Submitters
Logging
Local resources, Service grid resources, Desktop Grid resources, Web services, Databases
Resources: middleware service layer

7. gUSE application: Acyclic dataflow
Job to run on dedicated machine
Job to run in a gLite VO
Job to run in a Globus 2 VO
Job to run in a Globus 4 VO
Task to run in a BOINC Grid
Web service invocation
Database operation (R / W)
File from the client host
File from a GridFTP site
File from an LFC catalog (content from gLite SE)
Input string for a task or service
Result of a Database query

8. Dataflow programming with gUSE
Separate application logic from data
Cross & dot product data-pairing
All-to-all vs. one-to-one pairing of data items
(Concept from Taverna)
Generator components: to produce many output files from 1 input file
Collector components: to produce 1 output file from many input files
Any component can be generator or collector
Conditional execution based on equality of data
Nesting, cycle, recursion 40 50 20
1000 40
Generator
Collector
5000 1
7042 tasks
Collector
5000 1

9. Task execution process
User action, external event or time triggering
WS-PGRADE
gUSE Web Services
Workflow storage
File storage
Workflow Engine
Meta-broker
Local Submitter
EGEE Submitter
Desktop Grid Submitter
Web Service Client
Database Client …
Dedicated cluster
gLite WMS
Desktop Grid server
Web Service
DBMS

10. gUSE Generic Grid-Grid bridge
EGEE VO 1
EDGeS
Desktop Grid
3G Bridge
DG server
WMS
Machine
Other EGEE
services
DC-API plugin
UI machine
3G Bridge
EGEE plugin
WS-PGRADE gUSE
UI machine
Comp. Element WN WN WN
EGEE VO 2 WN WN WN

11. Ergonomics Users can be grid application developers or end-users.
Application developers design sophisticated dataflow graphs in gUSE
embedding into any depth, recursive invocations, conditional structures, generators and collectors at any position
Use personal grid certificate for test executions
Publish applications in the repository at certain stages of work
Graphs
Templates
Concrete
End-users see gUSE as a science gateway
List ready to use applications from repository
Import and execute applications without knowledge of programming, dataflow, grid, internal structure of application
Use personal grid certificate for production execution

12. End users’ view

13. Import an application from repository
To avoid overwriting any of you existing applications:
Choose a new name!

14. notification
Ask notification to know when the execution of your application is finished!

15. Provide input for the application then submit!
Application list
Provide input for the application then submit!

16. Define when should WSPGRADE submit your application!
Set auto-submission
Define when should WSPGRADE submit your application!

17. Submission triggered by an external event
Your application will be submitted when a request arrives with the key that you just set. (WS invocation)

18. Write you own submission trigger!
URL of the WS-PGRADE portal server
pUser
Owner of the application (Portal User)
pID
Key to identify the call
pText
Freeform string to identify the application instance

19. Set custom input files/ input Strings/ SQL queries for components

20. Monitoring application execution
Step 1:
The application is selected by button “Submit”
Step 2:
Define a freeform description to identify this particular executable instance

21. States of a Workflow Instance
1.If all state counters are 0 then there is no Instance of the given Workflow
2. In Column “Error” the number instances being in states “Error” and “Aborted” are summed
3. Instances in state “Suspended” are displayed according their preceding states
Origin
Submitted
Running
Suspended
Aborted
Finished
Error
Submit
Suspend
Resume
Abort
First Job Starts
Last Job terminates
Internal Error

22. Keep large data in Grid files or DBMS Downloading results
Information about the quota of the user allotted storage capacity in the Portal server
Keep large data
in Grid files or DBMS
Columns of individual instances, please note, that outputs can be downloaded separately
Inner slider to encounter and access each Instances
Columns of bulk download: All or proper parts of all instances of a given WF can be downloaded

23. Application developers’ view

24. Development cycle of a gUSE dataflow application
Graph
Component layout I/O Ports Edges
Which parts and parameters can be modified, which cannot
gUSE Application repository service
Define content
Concrete Workflow
Algorithms/services Resources Inputs
Template
Constraints Comments
Repository Item
Prepare for re-usage
Publish
Import
Test execution
End user’s account
Concrete Workflow
Workflow Instance
Running state Outputs
Submit
Workflow instance
Workflow instance
Workflow instance

25. Graph editor (P-GRADE Portal look-and-feel)
Components remain empty at this stage!

26. Workflow Configuration: Creating Concrete Workflow
Note the inner slider: By moving it you can encounter –and make visible – any Port of the current job
Workflow Configuration: Creating Concrete Workflow
Select Configure
Select a job by mouse click
Fill the job property characteristics.
Details have discussed previously.
Select Port Property Configuration
Fill port property characteristics. Details have discussed previously.
Select JDL/RSL Configuration
Return to main view
Insert a definition
Select one of the JDL/RSL Configuration Parameters of the list box
Save & Upload the Workflow configuration. Remind eventual error messages!
Close the configuration of this job
Confirm the settings

27. Explaining Generator, Normal and Collector job types
Number of results generated by a single query can be unknown!
Generator
Collector *3
1 run
1 run 1 1 2
Typically for generating input parameters of a simulation. (Often a database query) 2
Typically for statistical analysis
1. run
2. run
3. run
Typically for processing data in “parameter sweep” fashion

28. Configuring the Workflow: Data processing pattern
Number of data elements
Define input data elements for every open input Port m n h
Determine Job to be Generator by defining Multiple output port. In this case the job may be able to produce more than 1 jobs associated to the multiple output port within one job submission step G *K
Determine Dot or Cross product relation of Input ports to define the number of job submissions 1
Legend:
Cross Product
Dot Product C
Determine Job to be Collector by defining a Gathering Input Port. The Job execution will be postponed until all input files to that Port have arrived and can be elaborated in a single job submission step

29. Animating the number of generated output data elements
In case of Generator job the number of job submissions may differ from the number of files on Output Ports m n h
m*n h G
In case of dot product the Job is submitted with input files having a common index number in each input Ports
m*n
h*K *K
m*n
h*K
m*n
h*K
m*n*h*K S
m*n*h*K S
S=max(m*n,h*k) 1 1 S S C S S S
In case of cross product individual Job submission is generated for each possible input file combination S S

30. Data processing complexity: P-GRADE or WS-PGRADE
P-GRADE Portal:
Generators only the first level of graph
Collectors always the last level of graph
Always CROSS product between parameter input channels
WS-PGRADE Portal:
Generators at any level of graph
Collectors at any level of graph
Freedom to use CROSS or DOT product G G G G G
Normal WF C C

31. Other advanced features for developers
Workflow embedding
Simplify workflows by hierarchical development
Special case – Iterative embedding: iterate computation towards the final result
Conditional execution of workflow branches
Execute a branch only if it receives the expected input

32. Example: Need for complexity CancerGrid workflow1
N=20e-30e, M=100  ~2.7 billion tasks !!! x1
NxM 1 xN xN xN
NxM
NxM 1 N N N
Generator job
Generator job
NxM

33. Current users of gUSE CancerGrid project
Predicting various properties of molecules to find anti-cancer leads
Creating science gateway for chemists
EDGeS project (Enabling Desktop Grids for e-Science)
Integrating EGEE with BOINC and XtremWeb technologies
User interfaces and tools
ProSim project
In silico simulation of intermolecular recognition
JISC ENGAGE program
University of Westminster Desktop Grid
Using AutoDock on institutional PCs

34. Conclusions P-GRADE Portal remains supported WS-PGRADE
Features can serve most grid scenarios
Open source project on Sourceforge
WS-PGRADE
Implemented on top of scalable, WS based gUSE architecture
More expressive dataflow patterns
Transparent access to
Local resources
Service Grids
Desktop Grids
Databases
Web services
Application repository
Service for collaboration of developers and end-users

35. Next steps with gUSE: www.guse.hu
User manual
Request an account

36. Thank you Gergely Sipos sipos@sztaki.hu www.wspgrade.hu www.guse.hu