1. The EDGeS project receives Community research funding 1 Bridging EGEE to BOINC and XtremWeb GIN : From interoperation to interoperability

2. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 2 From interoperation to interoperability Overview Bridging EGEE to BOINC and XtremWeb GIN : From interoperation to interoperability Overview Definitions : Service Grids and Desktop GridsDefinitions : Service Grids and Desktop Grids Presentation of the EDGeS projectPresentation of the EDGeS project Bridge BOINC  EGEEBridge BOINC  EGEE Bridge XtremWeb  EGEEBridge XtremWeb  EGEE Bridge EGEE  BOINCBridge EGEE  BOINC Bridge EGEE  XtremWebBridge EGEE  XtremWeb Architecture of the EDGeS 3G BridgeArchitecture of the EDGeS 3G Bridge Desktop Grid Production InfrastructureDesktop Grid Production Infrastructure OGF standards used for future interoperabilityOGF standards used for future interoperability

3. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 3 SG = Service Grid = Managed grid of managed computing clusters Bridging EGEE to BOINC and XtremWeb SG = Service Grid = Managed grid of managed computing clusters Grid User X509 proxy X509 proxy with VOMS extensions Submits Job with X509 proxy Publishes available Resources Pushes Job Log VOMS Admin Manages VO Site Admin Manages Site Accesses Data with X509 proxy Gives Job Status Gives Accounting and Auditing VOMS Server Accounting Logging & Bookkeeping Meta-scheduler (WMS) Site Computing Resource Site Storage Resource Grid Admin Sends back Output Sandbox

4. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 4 SG = Service Grid = Managed grid of managed computing clusters Bridging EGEE to BOINC and XtremWeb SG = Service Grid = Managed grid of managed computing clusters Computing and Storage Resources are managed by trained staff inside Sites and are authenticated by X509 certificates.Computing and Storage Resources are managed by trained staff inside Sites and are authenticated by X509 certificates. Users are authenticated by X509 certificates or proxies.Users are authenticated by X509 certificates or proxies. Users belong to VOs and get a X509 proxy from a VOMS server to : – Access data, – Submit jobs.Users belong to VOs and get a X509 proxy from a VOMS server to : – Access data, – Submit jobs. Executables are NOT authenticated.Executables are NOT authenticated. So trust is primarily between Sites and VOs. Order of magnitude is typically 100 000 CPUs.Order of magnitude is typically 100 000 CPUs. A meta-scheduler (WMS) pushes the jobs to resources with are both suitable and available.A meta-scheduler (WMS) pushes the jobs to resources with are both suitable and available. Examples : EGEE, NorduGrid, OSG, DEISA, …

5. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 5 DG = Desktop Grid = Loose grid scavenging idle resources Bridging EGEE to BOINC and XtremWeb DG = Desktop Grid = Loose grid scavenging idle resources Unit of Work = Application + Input Data Unit of Work = Application + Input Data Grid User Submits input data for an application Requests Unit of Work Sends Unit of Work Application Manager Certifies Application Resource Owner (often volunteer) Owns Resource Sends back results Accepts or Refuses an application on his resource Grid Server with Application Repository Computing Resource (often Desktop Computer) Sends back results Currently, for BOINC, both roles of ‘Application Manager’ and ‘Grid User’ are fulfilled by ‘BOINC Project Owners’.

6. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 6 DG = Desktop Grid = Loose grid scavenging idle resources Bridging EGEE to BOINC and XtremWeb DG = Desktop Grid = Loose grid scavenging idle resources Computing and Storage Resources are owned by various Owners (it is often volunteer computing), but they are NOT managed and NOT authenticated.Computing and Storage Resources are owned by various Owners (it is often volunteer computing), but they are NOT managed and NOT authenticated. Grid Servers are authenticated by a X509 certificate.Grid Servers are authenticated by a X509 certificate. Users are authenticated by the Grid Servers, but NOT by the Computing and Storage Resources.Users are authenticated by the Grid Servers, but NOT by the Computing and Storage Resources. Executables are certified by managers of the Grid Servers.Executables are certified by managers of the Grid Servers. So :– Resource Owners have to trust the Grid Servers, – BOINC sends each Work Unit to several Resource Owners, because BOINC does NOT fully trust them. Order of magnitude can be 1 000 000 CPUs.Order of magnitude can be 1 000 000 CPUs. Starving Computing Resources pull Work Units from Grid Servers.Starving Computing Resources pull Work Units from Grid Servers. Examples : BOINC, XtremWeb, xGrid

7. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 7 Presentation of the EDGeS project Bridging EGEE to BOINC and XtremWeb Presentation of the EDGeS project New FP7 project started on 01/01/2008 Integrate Service Grids and Desktop GridsIntegrate Service Grids and Desktop Grids Enable very large number of computing resources (100K-1M processors)Enable very large number of computing resources (100K-1M processors) Attract new scientific communitiesAttract new scientific communities Provide a Grid application development environmentProvide a Grid application development environment Provide application repository and bridges for the execution in the SG-DG systemProvide application repository and bridges for the execution in the SG-DG system WLCG (CERN) EDGeS gLite (EGEE) ARC (NorduGrid) Boinc (Berkeley) XtremWeb (INRIA/IN2P3) Xgrid (Apple) Unicore (DEISA) VDT (OSG) Current Future

8. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 8 Presentation of the EDGeS project Bridging EGEE to BOINC and XtremWeb Presentation of the EDGeS project http://www.edges-grid.eu Now, Interoperation : Ad-hoc bridges and interfaces between EGEE, BOINC and XtremWeb.Ad-hoc bridges and interfaces between EGEE, BOINC and XtremWeb. A MoU between EDGeS and EGEE has been signed on 23 Sept 2008.A MoU between EDGeS and EGEE has been signed on 23 Sept 2008. XtremWeb users must have a X509 certificate, be registered in a VO and submit their Jobs with a X509 proxy.XtremWeb users must have a X509 certificate, be registered in a VO and submit their Jobs with a X509 proxy. BOINC Project Owners must have a X509 certificate, be registered in a VO and store a medium-term X509 proxy in a MyProxy server.BOINC Project Owners must have a X509 certificate, be registered in a VO and store a medium-term X509 proxy in a MyProxy server. All files must be transferred through the Input and Output sandboxes.All files must be transferred through the Input and Output sandboxes. In the future : Interoperability using OGF standards, in order to bridge more Grids.Interoperability using OGF standards, in order to bridge more Grids. Better support of grid file access (GFAL, lcg_utils and GridFTP).Better support of grid file access (GFAL, lcg_utils and GridFTP).

9. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 9 EGEE WMS EDGeS 3G bridge EGEE Plugin 1 for each (BOINC Project Owner, EGEE VO) pair Queue Manager & Job DB BOINC Handler 1 for each (BOINC server, BOINC Project Owner, EGEE VO) triple Bridge BOINC  EGEE (WU = Work Unit) Bridging EGEE to BOINC and XtremWeb Bridge BOINC  EGEE (WU = Work Unit) WU i+1 WU i+2 WU i+3 Job i+1 Job i+2 BOINC Server Work Unit BOINC Project Owner Submission MyProxy Medium term X509 proxy Config. file Credential access information Short term X509 proxy VOMS Server VOMS extensions Job Handler Interface Grid Handler Interface BOINC jobwrapper client (simulating a large BOINC computing resource) 3G job- wrapper

10. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 10 Bridge BOINC  EGEE Bridging EGEE to BOINC and XtremWeb Bridge BOINC  EGEE Solution = Inside EDGeS bridge, marshalling of the BOINC Work Units into Job collections For each (BOINC server, BOINC Project Owner, EGEE VO) triple, a separate Job Handler collects the BOINC Work Units and place them in a queue.For each (BOINC server, BOINC Project Owner, EGEE VO) triple, a separate Job Handler collects the BOINC Work Units and place them in a queue. For each (BOINC Project Owner, EGEE VO) pair, a separate EGEE plugin :For each (BOINC Project Owner, EGEE VO) pair, a separate EGEE plugin : –Retrieves a short term X509 Proxy for the BOINC Project Owner from a MyProxy server, and VOMS extensions from a VOMS server, –Periodically processes new Work Units found in the queue : It converts each Work Unit into an EGEE Job, In order to reduce the usage of the EGEE WMS, it uses Collection possibili- ties of EGEE to submit many Jobs in one request described using JDL.

11. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 11 EGEE Bridge XtremWeb  EGEE Bridging EGEE to BOINC and XtremWeb Bridge XtremWeb  EGEE XtremWeb User X509 proxy X509 proxy with VOMS extensions Submits User Job with X509 proxy Sends back Job Status and Results VOMS Server XtremWeb Server Submits mono-user Pilot Job with X509 proxy Gives Pilot Job Status gLite WMS Computing Element Pushes Pilot job Mono-user Pilot Job Requests only 1 User Job Sends 1 User Job with same X509 proxy User Job Gives Pilot Job Status Sends back results directly XtremWeb Bridge Requests User Jobs Sends User Jobs with X509 proxy Manages User Job status

12. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 12 Bridge XtremWeb  EGEE Bridging EGEE to BOINC and XtremWeb Bridge XtremWeb  EGEE Solution = XtremWeb bridge : Gliding with a mono-user Pilot Job 1.A XtremWeb User submits to the XtremWeb server his User Job with a X509 proxy. 2.At the request of the XtremWeb bridge, the XtremWeb server sends him the User Job with the X509 proxy. 3.The XtremWeb bridge submits to a gLite WMS a mono-user Pilot Job with this X509 proxy (job description in a JDL). 4.The gLite WMS pushes the Pilot Job to a Computing Element, which executes it. 5.The mono-user Pilot Job requests 1 User Job from the XtremWeb server, and stops itself if it receives none. 6.The XtremWeb server verifies that the requested User Job has a X509 proxy, and sends the User Job and the X509 proxy to the Pilot Job. 7.The Pilot Job verifies that the received X509 proxy is the same as its own X509 proxy, and executes the User Job. 8.At the end of the User Job, the Pilot Job sends the Job results directly to the XtremWeb server, then stops itself.

13. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 13 BOINC Server EGEE LCG-CE for EDGeS EDGeS 3G bridge Bridge EGEE  BOINC Bridging EGEE to BOINC and XtremWeb Bridge EGEE  BOINC Gets EXE Reports resources and performance Checks EXE Adds job Watches job X509 proxy with VOMS extensions BOINC Computing Resource BOINC plugin (DC-API) EDGeS Application Repository Information provider GRAM Job Manager for EDGeS EGEE VOMS EGEE User Queue Manager & Job DB Generic Job WS Handler BOINC Service Watches Pushes job Submits Job Logs events EGEE BDII gLite WMS EGEE LB Sends output Gets output

14. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 14 Bridge EGEE  BOINC Bridging EGEE to BOINC and XtremWeb Bridge EGEE  BOINC Solution = Installation of a LCG-CE sending the EGEE Jobs to the EDGeS bridge marshalling them into BOINC Works Units Publish information to the BDII according to GLUE 1.3Publish information to the BDII according to GLUE 1.3 EGEE producerEGEE producer –New GRAM job manager –Gets job information from wrapper –Checks if exe is validated in the EDGeS application repository (GEMLCA) –Checks if exe is supported by attached BOINC –Gets files from WMS –Adds job to 3G bridge job Database –Polls status of jobs in 3G bridge job Database –Gets results from 3G bridge and uploads to LB BOINC plugin (DC-API) BOINC plugin (DC-API) –Use DC-API to generate BOINC WUs –Jobs are read from the 3G bridge DB –3G DB entries are updated on events –The plugin has already been implemented for the CancerGrid system

15. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 15 EGEE LCG-CE for XtremWeb XtremWeb Server Bridge EGEE  XtremWeb Bridging EGEE to BOINC and XtremWeb Bridge EGEE  XtremWeb Solution Inside a LCG-CE, installation of a GRAM jobmanager to marshal the EGEE Jobs into XtremWeb Jobs. Gets EXE Watches Reports resources and performance Pushes job Checks EXE Adds job Watches job Submits Job Logs events X509 proxy with VOMS extensions Logs events XtremWeb Computing Resource EDGeS Application Repository EGEE BDII gLite WMS Information provider GRAM Job Manager for XtremWeb EGEE LB EGEE VOMS EGEE User Sends output Gets output

16. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 16 EDGeS 3G Bridge EGEE Architecture of the EDGeS 3G Bridge Bridging EGEE to BOINC and XtremWeb Architecture of the EDGeS 3G Bridge Job Handler Interface Job Database Queue Manager GridHandler Interface BOINC Plugins (DC-API) EGEE Plugins Scheduler Handler for received jobs Storage for received jobs Generic interface above grid plugins Grid plugin (submit jobs, update status, get output,...)‏ Control path WU  Job BOINC Handlers EGEE Handler Generic handler for received jobs LCG-CE for EDGeS gLite WMS BOINC Server User Job with X509 proxy Work Unit BOINC Server gLite WMS WU Job Control path Job  WU Job WU XtremWeb Plugins XtremWeb Server

17. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 17 XtremWeb Desktop GridsBOINC Desktop Grids Desktop Grid Production Infrastructure Bridging EGEE to BOINC and XtremWeb Desktop Grid Production Infrastructure EGEE EDGeS 3G bridge Local DG UoW Grid 1.500 PCs Public DG Extremadura Grid 70.000 PCs Public DG EGEE@home Planned 10.000 PCs Public DG SZDG 30.000 PCs Public DG AlmereGrid 3.000 PCs  Public DG EGEE  XtremWeb 1.000 PCs Public DG INRIA Grid 300 PCs Local DG IN2P3 Grid 200 PCs EGEE User gLite WMS BOINC plugin (DC-API) LCG-CE for BOINC LCG-CE for XtremWeb Job

18. Authors :Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah v2.13 18 OGF standards used for future interoperability Bridging EGEE to BOINC and XtremWeb OGF standards used for future interoperability GLUE 2.0 in order to publish information to the BDII : Needs implementation by gLite.GLUE 2.0 in order to publish information to the BDII : Needs implementation by gLite. BES to receive Job submissions : For example from GridSphere Portal.BES to receive Job submissions : For example from GridSphere Portal. BES to submit Jobs : Needs availability of CREAM CE.BES to submit Jobs : Needs availability of CREAM CE. JSDL to describe Jobs : Needs implementation by gLite.JSDL to describe Jobs : Needs implementation by gLite. Potentially : AUTHZ for Authentication / AuthorizationAUTHZ for Authentication / Authorization UR, RUS for Job logging and accountingUR, RUS for Job logging and accounting ByteIO, SRM, GridFTP, DMI to manage data transfersByteIO, SRM, GridFTP, DMI to manage data transfers ACS for the GEMLCA application repositoryACS for the GEMLCA application repository SAGA, DRMAA for the methodology of application developmentSAGA, DRMAA for the methodology of application development