8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 DEGISCO Desktop Grids for International Scientific Collaboration Peer-to-Peer Desktop Grids Francisco Brasileiro Universidade Federal de Campina Grande, Brazil Department of Systems and Computing Distributed Systems Lab 1 RI DEGISCO is supported by the FP7 Capacities Programme under contract nr RI

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 How to choose a desktop grid? This is the kind of question that the IDGF may help you to answer This is the kind of question that the IDGF may help you to answer Of course, there is no silver bullet Of course, there is no silver bullet Different ways to gather the desktops that form the grid Different ways to gather the desktops that form the grid Different workloads Different workloads Different life cycles for applications Different life cycles for applications Different system requirements Different system requirements … 2

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Resource provision in desktop grids Home users’ desktops Home users’ desktops Very large scale Very large scale High dispersion leads to a higher cost to attract volunteers High dispersion leads to a higher cost to attract volunteers Organization’s desktops Organization’s desktops Scale depends on the size of the organization Scale depends on the size of the organization “Attract volunteers” is a matter of organizations’ policy on the use of surplus capacity of their desktops “Attract volunteers” is a matter of organizations’ policy on the use of surplus capacity of their desktops 3

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Typical workloads of desktop grids Very large workloads (at least dozens of millions of hours of CPU) Very large workloads (at least dozens of millions of hours of CPU) They run from a few weeks to a few months in a large volunteer computing infrastructure They run from a few weeks to a few months in a large volunteer computing infrastructure Large workloads (from hundreds of thousands to a few millions of hours of CPU) Large workloads (from hundreds of thousands to a few millions of hours of CPU) They run in no longer than a week in a large campus desktop grid They run in no longer than a week in a large campus desktop grid Sizeable workloads (a few thousands of hours of CPU) Sizeable workloads (a few thousands of hours of CPU) Cannot be run on a single machine, but would run on the desktops of a small research lab Cannot be run on a single machine, but would run on the desktops of a small research lab However, the faster they run, the better! However, the faster they run, the better! Experiments with different parameters are often repeated Experiments with different parameters are often repeated 4

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Desktop grid applications and their users’ distribution 5 Applications Number of users Porting the applications to the execution infrastructure needs to be simple in the long tail, so that users themselves do the job

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Desktop grid applications and their life times’ distribution 6 Applications Life time Setting up the infrastructures for running the applications in the long tail needs to be cheap and fast, so to be affordable and useful

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 P2P desktop grids are most useful for … Medium and small labs Medium and small labs They don’t have much resources of their own They don’t have much resources of their own They might not have the funds and/or appeal to market a volunteer computing project They might not have the funds and/or appeal to market a volunteer computing project Sizeable workloads Sizeable workloads In P2P systems, every new resource provider is also a resource consumer In P2P systems, every new resource provider is also a resource consumer Applications in the long tail Applications in the long tail Cost of deploy is shared by all labs (peers) Cost of deploy is shared by all labs (peers) Security model requires virtualization which has the side effect of facilitating porting Security model requires virtualization which has the side effect of facilitating porting 7

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Rational of a peer-to-peer grid Each lab corresponds to a peer in the system and contributes with its idle resources Each lab corresponds to a peer in the system and contributes with its idle resources Users want to reduce the makespan of their applications and shorten their research cycle Users want to reduce the makespan of their applications and shorten their research cycle cpu utilization for lab 1 100% real time cpu utilization for lab 2 100% real time cpu utilization for the joint p2p grid 100% real time 8

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Volunteer vs p2p computing In a volunteer computing project, resource providers (the volunteers) and resource consumers (the scientists) are different people In a volunteer computing project, resource providers (the volunteers) and resource consumers (the scientists) are different people In a P2P grid, resource providers and resource owners are the same people (scientists) In a P2P grid, resource providers and resource owners are the same people (scientists) In a volunteer computing project, resources are donated, because volunteers think that they are going to be used for the good of mankind In a volunteer computing project, resources are donated, because volunteers think that they are going to be used for the good of mankind In P2P grids, idle resources are donated, because each resource owner wants to be able to use the others’ resources in the future, when he/she needs them In P2P grids, idle resources are donated, because each resource owner wants to be able to use the others’ resources in the future, when he/she needs them 9

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 OurGrid in a nutshell 10 User interface, job monitoring and scheduling Security and local sharing policy enforcement Site level and inter-site management of workers Rendezvous service

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 OurGrid proposition for a P2P grid Labs can freely join the system without any human intervention Labs can freely join the system without any human intervention No need for negotiation; no paperwork No need for negotiation; no paperwork Clear incentive to join the system Clear incentive to join the system One cannot be worse off by joining the system One cannot be worse off by joining the system Noticeable increased response time Noticeable increased response time Free-riding resistant Free-riding resistant Basic dependability properties Basic dependability properties Configurable level of security Configurable level of security Resilience to faults Resilience to faults Scalability Scalability Easy to install, configure, manage and program Easy to install, configure, manage and program No need for specialized support team No need for specialized support team 11

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 But there is no free lunch To simplify the problem, OurGrid focuses on a subset of Bag-of-Tasks applications To simplify the problem, OurGrid focuses on a subset of Bag-of-Tasks applications Communication with the outside world is not allowed Communication with the outside world is not allowed Facilitates scheduling and security enforcement Facilitates scheduling and security enforcement Simple fail-over/retry mechanisms to tolerate faults Simple fail-over/retry mechanisms to tolerate faults No need for QoS guarantees No need for QoS guarantees 12

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Current status of the OurGrid Community 13

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 How to start using OurGrid? 14 Using the public portal at Using the public portal at

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Once logged in, select an specific application or run a generic job 15

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Once a job is started, you can monitor its execution 16

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Keep track of your files with the file manager application 17

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 OurGrid, DEGISCO and IDGF OurGrid is promoted by the following documents and tools by DEGISCO and IDGF (among others): OurGrid is promoted by the following documents and tools by DEGISCO and IDGF (among others): D2.1.2: Policies and setup operations for Grid operators (update) D2.1.2: Policies and setup operations for Grid operators (update) D2.4.1: Report on best practices in infrastructure support D2.4.1: Report on best practices in infrastructure support D3.1.2: Identification and analysis of potential applications to be ported to the production DCI platform (update) D3.1.2: Identification and analysis of potential applications to be ported to the production DCI platform (update) A Road Map: Desktop Grids for eScience A Road Map: Desktop Grids for eScience The OurGrid support system acts as a 2nd level helpdesk of DEGISCO infrastructure (based on Request Tracker) The OurGrid support system acts as a 2nd level helpdesk of DEGISCO infrastructure (based on Request Tracker) 18

8 th International Desktop Grid Federation Workshop, Hannover, Germany, August 17 th, 2011 Thanks for your attention! IDGF web site (desktopgridfederation.org) IDGF web site (desktopgridfederation.org)desktopgridfederation.org OurGrid Web Site ( OurGrid Web Site ( OurGrid Community (status.ourgrid.org) OurGrid Community (status.ourgrid.org)status.ourgrid.org Francisco Brasileiro Francisco Brasileiro 19 Contacts: