1. The EDGeS project receives Community research funding 1 SG-DG Bridges Zoltán Farkas, MTA SZTAKI

2. Presentation title Author: SG-DG Bridges Zoltán Farkas Outline Introduction, aimsIntroduction, aims SG features: EGEESG features: EGEE DG features: BOINC (and XtremWeb)DG features: BOINC (and XtremWeb) BOINC -> EGEE bridgeBOINC -> EGEE bridge 3G Bridge architecture3G Bridge architecture EGEE -> BOINC bridgeEGEE -> BOINC bridge

3. Presentation title Author: SG-DG Bridges Zoltán Farkas Introduction, aims The EDGeS project aims to offer an infrastructure that integrates Service Grid (SG) and Desktop Grid (DG) infrastructuresThe EDGeS project aims to offer an infrastructure that integrates Service Grid (SG) and Desktop Grid (DG) infrastructures Users of one grid type should be able to make use of the other grid type in a transparent way and vice versaUsers of one grid type should be able to make use of the other grid type in a transparent way and vice versa Thus, the integrated infrastructure will offer the advantages of the two grid typeThus, the integrated infrastructure will offer the advantages of the two grid type The core component of this infrastructure is the SG-DG bridge technologyThe core component of this infrastructure is the SG-DG bridge technology

4. Presentation title Author: SG-DG Bridges Zoltán Farkas 4 SG features – EGEE I. A big set of services:A big set of services: WMS: broker, scheduling jobs to resources LB: logging and bookkepping service BDII: information system WN: worker node, does actual job execution CE: computing element, collects WNs in a queue using an LRMS SE: storage element, used to store large files LFC: file catalogue, files stored on SE can be organized into a directory structure MyProxy: proxy certificate storage VOMS: virtual organization membership handling component R-GMA & APEL: accounting services

5. Presentation title Author: SG-DG Bridges Zoltán Farkas 5 SG features – EGEE II. Mostly institutes provide the computing resourcesMostly institutes provide the computing resources Resources are organized into Virtual OrganizationsResources are organized into Virtual Organizations Users with a registered certificate accepted in some VO can use the infrastructureUsers with a registered certificate accepted in some VO can use the infrastructure Basically any kind of job can be executed, with some restrictionsBasically any kind of job can be executed, with some restrictions

6. Presentation title Author: SG-DG Bridges Zoltán Farkas 6 SG features – EGEE III.

7. Presentation title Author: SG-DG Bridges Zoltán Farkas 7 Grid features – BOINC I. One central service per project with limited access that stores work to be processedOne central service per project with limited access that stores work to be processed Desktop PCs connect with a simple client application and offer their free CPU cyclesDesktop PCs connect with a simple client application and offer their free CPU cycles Client application fetches workunits, processes them, and uploads results to the serverClient application fetches workunits, processes them, and uploads results to the server Mostly the same application is run with many input data sets (parameter study applications)‏Mostly the same application is run with many input data sets (parameter study applications)‏

8. Presentation title Author: SG-DG Bridges Zoltán Farkas 8 Grid features – BOINC II. BOINC project server BOINC Project Admin BOINC Client pool WU/APP/ Result Database Scheduler Work gen. Assim....

9. Presentation title Author: SG-DG Bridges Zoltán Farkas 9 BOINC -> EGEE I. Task to be solved:Task to be solved: Process BOINC workunits In the EGEE infrastructure Develop a bridge that:Develop a bridge that: Can handle BOINC workunits And is able to create EGEE jobs from the workunits, and run them in EGEE

10. Presentation title Author: SG-DG Bridges Zoltán Farkas 10 BOINC -> EGEE Possible solutions Agent-based execution:Agent-based execution: Send BOINC clients to EGEE BOINC client connects to BOINC server to fetch work and report results Wrapping workunits execution:Wrapping workunits execution: Send BOINC applications to EGEE Fetch BOINC workunits, and execute them in an EGEE job, finally report results

11. Presentation title Author: SG-DG Bridges Zoltán Farkas 11 BOINC -> EGEE First version EGEE UI Machine gLite WMS Boinc Client WU slot N WU slot 1 JobWrapper Process 1 fork() ‏  Create JDL file,  Submit,  Get status,  Get output. JobWrapper Process N fork() ‏ Watch Job i+1 Job i+N  Create JDL file,  Submit,  Get status,  Get output. EGEE CE WN EGEE CE WN

12. Presentation title Author: SG-DG Bridges Zoltán Farkas 12 Lessons learnt EGEE WMS doesn't like periodic interaction (needs restart every x hours)‏EGEE WMS doesn't like periodic interaction (needs restart every x hours)‏ Workunits should be gathered EGEE Operations sometimes failEGEE Operations sometimes fail If there is a failure, retry the operation at most three times Ways to improve:Ways to improve: Interact with WMS the least possible times Handle pack of jobs instead of individual jobs

13. Presentation title Author: SG-DG Bridges Zoltán Farkas 13 Improved BOINC → EGEE bridge Collect jobs originating from BOINC:Collect jobs originating from BOINC: Place them in a queue New jobs in the queue are periodically handled by an EGEE plugin, that Uses Collection possibilities of EGEE to submit many jobs in one request This way the usage of the WMS is reducedThis way the usage of the WMS is reduced

14. Presentation title Author: SG-DG Bridges Zoltán Farkas 14 Improved bridge architecture JobWrapper Process 1 JobWrapper Process N WU DB WU i+1 WU i+N WU i+2 WU i+3  Add  Check  Get output  Add  Check  Get output gLite WMS Job i+1 Job i+k EGEE Plugin Job i+1 Job i+2 Job i+k

15. Presentation title Author: SG-DG Bridges Zoltán Farkas 15 Bridge generalisation Jobwrapper → Source grid producersJobwrapper → Source grid producers Produce jobs originating from source grids WU DB → Job database + Queue ManagerWU DB → Job database + Queue Manager Stores job produced by source grid producers Selects jobs for execution EGEE plugin → Destination grid consumers/pluginsEGEE plugin → Destination grid consumers/plugins Execute jobs in the job database in the supported destination grids

16. Presentation title Author: SG-DG Bridges Zoltán Farkas 16 Generic Grid-Grid Bridge (3G Bridge)‏ Job Database + Queue Manager Src Grid Prod 1 Dst Grid 1 Dst Grid Cons 1 Src Grid 2 Src Grid Prod 2 Src Grid n Src Grid Prod n Dst Grid 2 Dst Grid Cons 2 Dst Grid m Dst Grid Cons m Src Grid 1

17. Presentation title Author: SG-DG Bridges Zoltán Farkas 17 Job Database + Queue Manager Job Handler Interface Job Database Queue Manager GridHandler Interface DC-API Plugin XWeb Plugin EGEE Plugin Scheduler Interface for sources Received job storage Received job handler, grid plugin user Generic interface above grid plugins Grid plugin (submit jobs, update status, get output,...)‏ Control path

18. Presentation title Author: SG-DG Bridges Zoltán Farkas 18 EGEE -> BOINC I. Transparent method for running EGEE jobs on BOINC DGsTransparent method for running EGEE jobs on BOINC DGs User interacts with EGEE using EGEE toolsUser interacts with EGEE using EGEE tools 3G Bridge used to transfer jobs to BOINC3G Bridge used to transfer jobs to BOINC Special CE created to catch EGEE jobsSpecial CE created to catch EGEE jobs EDGeS AR is used to check validity of applicationsEDGeS AR is used to check validity of applications

19. Presentation title Author: SG-DG Bridges Zoltán Farkas 19 EGEE job submission

20. Presentation title Author: SG-DG Bridges Zoltán Farkas 20 EGEE -> BOINC extension key concept Create a new GRAM jobmanager/LRMS:Create a new GRAM jobmanager/LRMS: –For every job, we get the job info (executable name, input files used) from the wrapper script submitted by the EGEE WMS –Add the job to the 3G Bridge –Report logging using DGAS/glite-lb-logevent –The 3G Bridge uses a DC-API plugin to run the job on BOINC

21. Presentation title Author: SG-DG Bridges Zoltán Farkas 21 3G Bridge: EGEE → BOINC Job Database + Queue Manager EGEE EGEE producer BOINCDC-API

22. Presentation title Author: SG-DG Bridges Zoltán Farkas 22 EGEE producer Overview A new GRAM jobmanagerA new GRAM jobmanager Gets job information from the WMS wrapper scriptGets job information from the WMS wrapper script Checks if exe is a validated oneChecks if exe is a validated one Checks if exe is supported by one of the attached BOINC (or XtremWeb) projectsChecks if exe is supported by one of the attached BOINC (or XtremWeb) projects Gets files from WMSGets files from WMS Adds job to 3G Bridge job DBAdds job to 3G Bridge job DB Polls status of jobs in 3G Bridge DBPolls status of jobs in 3G Bridge DB Gets results from 3G Bridge and uploads to WMSGets results from 3G Bridge and uploads to WMS

23. Presentation title Author: SG-DG Bridges Zoltán Farkas 23 DC-API plugin Use DC-API to generate BOINC WUsUse DC-API to generate BOINC WUs Jobs are read from the 3G bridge DBJobs are read from the 3G bridge DB 3G DB entries are updated on events3G DB entries are updated on events The plugin has already been implemented for the CancerGrid systemThe plugin has already been implemented for the CancerGrid system

24. Presentation title Author: SG-DG Bridges Zoltán Farkas 24 EGEE -> BOINC: Overview of the system EGEE WMS BOINC CE EGEE UI EDGeS Application Repository Get EXE Watch Get output LRMS EGEE BDII Info provider Report resources and performance Submit job Check EXE Add job Watch job BOINC BOINC client pool Submit job EGEE LB Log events Send output EGEE VOMS X509 proxy Log events 3G Bridge DC-API plugin DB

25. Presentation title Author: SG-DG Bridges Zoltán Farkas 25 3G Bridge Data Handling Issues EGEE applications might use huge input filesEGEE applications might use huge input files For data distribution, ADICS/ATTIC can be used (developed by Cardiff University)For data distribution, ADICS/ATTIC can be used (developed by Cardiff University) 3G Bridge uses ATTIC to publish selected files (recent development)3G Bridge uses ATTIC to publish selected files (recent development) ATTIC support in DGs (BOINC/XtermWeb) is work in progressATTIC support in DGs (BOINC/XtermWeb) is work in progress

26. Presentation title Author: SG-DG Bridges Zoltán Farkas 26Conclusions The 3G Bridge architecture:The 3G Bridge architecture: –Offers transparent way for running jobs on BOINC for EGEE users –Offers transparent way for running BOINC jobs on the EGEE infrastructure Has been extended to support P-GRADE Portal parameter study applications (thus special case of remote file handling is solved)Has been extended to support P-GRADE Portal parameter study applications (thus special case of remote file handling is solved) Initial support for handling large amount of dataInitial support for handling large amount of data Existing 3G Bridge plugins: EGEE, DC-API, XtremWebExisting 3G Bridge plugins: EGEE, DC-API, XtremWeb Future 3G Bridge plugins: OurGridFuture 3G Bridge plugins: OurGrid