WS-PGRADE portal and its usage in the CancerGrid project M. Kozlovszky, P. Kacsuk Computer and Automation Research Institute of the Hungarian Academy of Sciences PUCOWO, Zurich, Switzerland 10-11/06/2010

2 Motivations of creating gUSE To overcome (most of) the limitations of P-GRADE portal: To provide better modularity  to replace any service To improve scalability  to millions of jobs To enable advanced dataflow patterns To interface with wider range of resources To separate Application Developer view from Application User view WS-PGRADE (Web Services Parallel Grid Runtime and Developer Environment) and gUSE (Grid User Support Environment) architecture

3 WS-PGRADE/gUSE Creating complex workflow and parameter sweeps Seamless access to various types of resources clusters, service grids, desktop grids, databases. Scalable architecture Advanced data-flows Creating complex applications using embedded workflows, legacy codes Comfort features Separated views Community components from workflow repository

4 WS-PGRADE architecture Graphical User Interface: WS-PGRADE Workflow Engine Workflow storage File storage Application repository Logging gUSE information system Submitters Gridsphere portlets Autonomous Services: high level middleware service layer Resources: middleware service layer Local resources, service grid VOs, Desktop Grid resources, Web services, Databases gUSE Meta-broker Submitters File storage Submitters

5 Application lifecycle in WS-PGRADE Define workflow structure Configure workflow Define content for tasks Run a test Use local resources, Web services, Databases Scale workflow for large simulations Use batch systems, use cluster grids, use desktop grids Fix some parameters, leave some open Result: An application specific science gateway for end users

6 WS-PGRADE application: Acyclic dataflow Job to run on dedicated machine Job to run in a gLite VO Job to run in a Globus 2/4 VO Task to run in a BOINC Grid Web service invocation Database operation (R / W) File from the client host File from a GridFTP site File from an LFC catalog Input string from a task or service Result of a Database query

7 Dataflow programming with gUSE Separate application logic from data Cross & dot product data-pairing Concept from Taverna All-to-all vs. one-to-one pairing of data items Generator components: to produce many output files from 1 input file Collector components: to produce 1 output file from many input files Any component can be generator or collector Conditional execution based on equality of data Nesting, cycle, recursion tasks

8 Ergonomics Users can be grid application developers or end-users. Application developers design sophisticated dataflow graphs Embedding into any depth, recursive invocations, conditional structures, generators and collectors at any position Publish applications in the repository at certain stages of work Applications Projects Concrete workflows Templates Graphs End-users see WS-PGRADE & gUSE as a science gateway List of ready to use applications in repository Import and execute application without knowledge of programming, dataflow or grid

9 Current users of gUSE EDGeS project (Enabling Desktop Grids for e-Science)EDGeS Integrating EGEE with BOINC and XtremWeb technologies User interfaces and tools ProSim projectProSim In silico simulation of intermolecular recognition See next presentation University of Westminster Desktop GridUniversity of Westminster Desktop Grid Using AutoDock on institutional PCs CancerGrid projectCancerGrid Predicting various properties of molecules to find anti-cancer leads Creating science gateway for chemists

10 Motivation to use gUSE and WS-PGRADE for CancerGrid Arbitrary number of generators (and collectors) within one workflow (at arbitrary locations). Scalability: Number of jobs within one workflow is at range: 100K…1M ! Import of the existing EndUser configuration GUI (Easy-to- use, web based, user specific) application specific portlet for end users was not needed.

11 molecule database Executing workflows Browsing molecules DG clients from all partners Molecule database server Portal and DesktopGrid server BOINC server 3G Bridge Portal DG jobs WU 1 WU 2 WU N Job 1 Job 2 Job N GenWrapper for batch execution BOINC client Legacy Application Portal Storage Local Resource Local jobs Legacy Application WU X WU Y

12 CancerGrid Workflows Descriptor CalculationProperty Prediction Model building Screening

13 Working on the CancerGrid Portal – step-by-step Initial state: molecules/structures stored in DB, organised into lists User selects list of molecules/structures User selects/downloads a workflow from repository User configures the workflow to take the list as input User optionally updates parameters of the modules Submits workflow Optionally monitors the status When workflow finished, results are stored in the DB

14 Molecule lists

15 Molecule viewer

16 Downloading workflow from repository – End user view

17 Workflow configuration

18 List of workflows (Novice user view)

19 Status monitor – End user view

20 Job statuses of a workflow - Developer view

21 Conclusions WS-PGRADE: Implemented on top of scalable, WS based gUSE architecture More expressive dataflow patterns Transparent access to Local resources Service Grids Desktop Grids Databases Web services Application repository Service for collaboration of developers and end-users

22 Next steps at User manual Request a user account

23 Thank you for your attention! Questions? Acknowledgement: CancerGrid EU FP6 project (FP LIFESCTHTALTH-7)

24 Applications in CancerGrid Flexmol is an XML-based molecular language Molecule 2D/3D converter (Cmol3D) Molecule 3D conformation generator (Cmol3D) MOPAC (Molecular Orbital PACkage) is a semiempirical quantum chemistry program based on Dewar and Thiel's NDDO approximation Codessa Pro (Comprehensive Descriptors for Structural and Statistical Analysis) is a software suite for developing quantitative structure-activity/property relationships Matrix former QSAR Model builder Quantitative structure-activity relationship (QSAR) is the process by which chemical structure is quantitatively correlated with a well defined process, such as biological activity or chemical reactivity. (Chemical) Property Predictor File format converters (to integrate the previous tools into a workflow)

25 Cmol3D property settings

26 The CancerGrid infrastructure PRODUCTION system gUSE portal BOINC server (private desktopgrid with firewall and controlled donor access) Monitoring info 69 machines (AMRI 10, SZTAKI 56, UPF 2, UoJ 1) TEST system gUSE portal BOINC server (private desktopgrid with firewall and controlled donor access) Monitoring info Performance measurements: mols6 days~70 machines10 confs