P-GRADE and WS-PGRADE portals supporting desktop grids and clouds Peter Kacsuk MTA SZTAKI

2 Parameter study workflow execution support in P-GRADE portal GEN SEQ COLL SEQ Parameter sweep jobs: To be executed in Grids, Desktop Grids and Clouds Generates input parameter space Evaluates the results of the simulation P-GRADE Portal

3 Motivation: Local DG in Univ. of Westminster based on BOINC (SZTAKI Desktop Grid) New Cavendish Street576 nodes 2.Marylebone Campus559 nodes 3.Regent Street395 nodes 4.Wells Street31 nodes 5.Little Tichfield Street66 nodes 6.Harrow Campus254 nodes Total:1881 nodes Lifecycle of a node: 1.PCs basically used by students/staff 2.If unused, switch to Desktop Grid mode 3.No more work from DG server -> shutdown (green solution)

4 Generating DG applications To port an application to a BOINC system requires three steps: 1.Registering the application at the BOINC server 2.Creating the master code running on the server 3.Creating the client code In BOINC Step 2 and 3 require the modification of the original application and this is not trivial Using the gUSE/WS-PGRADE environment: –Step 1 by the DG system administrator –Step 2 and 3 are done nearly automatically without any code modification

5 Automatic generation of master and client code To facilitate Step 2 and 3 SZTAKI developed the DC-API (Distributed Computing API) as part of the SZTAKI Desktop Grid (SZDG) package DC-API can automatically generate WUs for PS jobs arriving from the workflow However, DC-API still requires the modification of the original application to create the client code SZTAKI has also developed Genwrapper a generic wrapper that can –eliminate the boincification of the code –automatically generates the client code without the modification of the original code

6 gUSE architecture and usage WS-PGRADE Workflow Engine Workflow storage File storage EGEE Submitter Dedicated site gUSE Services Meta-broker Desktop Grid Submitter Local Submitter Web Service Client Database Client WMS DG server Web Service DBMS User action, external event or time triggering

7 Connecting gUSE with DGs in CancerGrid DG Server BOINC Server Components BOINC Task DB Sche- duler Data server Queue Manager DC-API master WU Job Database (Description of Jobs: Apps, Args, I/O files) Job descr. gUSE (W orkflow enactor) gUSEDesktopGridgUSE-DG integration Scheduling policy Batch creation gUSE Storage gUSE WS Submitter WS- PGRADE (User IF) (WF repre- sentation ) BOINC client GenWrapper for batch execution DC-API cli Legacy Application BOINC client GenWrapper for batch execution DC-API cli Legacy Application gUSE 3G Bridge Submitter Job descr. gUSE Local Submitter

8 UoW Local DG Protein Molecule Simulation using AutoDock WS-PGRADE portal

9 PS job execution in Grids by P-GRADE P-GRADE Portal 3G Bridge Target Grid Plugins gLite Grid ARC Grid BOINC Grid EC2, Eucalyptus

10 P-GRADE portal → 3G Bridge P-GRADE Portal Files WSClient Tomcat Runtime 3G Bridge Job DB 3G Bridge Job DB Queue Manager Grid Handler Interface Grid Plugin 2 BOINC Plugin 3 Cloud Plugin n WSSubmitte r Grid Plugin 1 Download Manager Download Manager HTTP D Submit job Check status Get Output

11 Host A 3G-Bridge DC-API – Condor Plugin Cloud Plugin Queue 1 … Queue 2 … Condor Submitter/ Master Cloud Resource 2 (Condor Worker) Cloud Resource N (Condor Worker) … Scheduler P-Grade Portal Amazon/ Eucalyptus Cloud Interface Cloud Resource 1 (Condor Worker) Legend Information Command Job 1.Job is submitted from P-Grade Portal to the 3G-Bridge. 2.3G-Bridge submits the job to a Condor Cluster using the DC-API Condor Plugin (Queue 1). The cluster consists of workers running in the cloud. 3.The Scheduler keeps track of the number of jobs in the Condor cluster and of the number of the running Cloud Resources (workers). 4.If the cluster is overcommitted, the Scheduler starts new workers by submitting a job to the 3G-Bridge queue of the Cloud Plugin (Queue 2). 5.If the cluster under utilized, the Scheduler stops some workers (cloud resources) by cancelling some jobs in the Cloud Plugin Queue (Queue 2). P-GRADE portal supporting Clouds

12 Conclusions By the P-GRADE portal you can submit PS jobs to –Grids –Local clusters –BOINC desktop grids –Clouds By writing new target DG plugins you can easily add new type of DG resources By writing new target Cloud plugins you can easily add new type of Cloud resources

Thank you