RI WP Asia 1-EPIKH School for Application Porting 1 Ловаш Р (MTA SZTAKI). Афанасьев А.П. (ИСА РАН) Волошинов В.В. ((ИСА РАН) Посыпкин М.А. (ИСА РАН) Сухорослов О.В. (ИСА РАН) Храпов Н.П. (ИСА РАН) DEGISCO is supported by the FP7 Capacities Programme under contract nr RI EXPANDING SCIENTIFIC COMPUTATIONAL INFRASTRUCTURES WITH DESKTOP GRIDS

RI WP Asia 1-EPIKH School for Application Porting 2 Desktop Grids can be categorized as a successful type of distributed infrastructure that is widely used in science and industry; both the largest Grids for science and in industry are based on this technology. The presentation provides an overview of Desktop Grids and their place in the infrastructure world. Desktop Grids introduction

RI WP Asia 1-EPIKH School for Application Porting 3 RI WP4 version: 1.0 Introduction Author: Robert Lovas, Ad Emmen Why Desktop Grids are important?

RI WP Asia 1-EPIKH School for Application Porting 4 Survey of EDGeS project Questionnaires all across Europe Get an idea of the interest in people and SMEs to donate computing time for science to a Grid Get an idea of the interest in running a Grid inside an SME Prelude - what do people at home and SME’s think about grid computing

RI WP Asia 1-EPIKH School for Application Porting 5 Survey amongst the General Public and SME’s

RI WP Asia 1-EPIKH School for Application Porting Opinions about Grid computing

RI WP Asia 1-EPIKH School for Application Porting 7 Overall: there is interest in Desktop Grid computing in Europe. However, that people are willing to change their current practice and say that they want to participate in Grid efforts does not mean that they are actually going to do that. Need to generate trust in the organization that manages the Grid. People want to donate computing time for scientific applications, especially medical applications. They do not like to donate computing time to commercial or defense applications. People want feedback on the application they are running. No clear technical barriers perceived by the respondents: so this does not need much attention. Overall the respondents were rather positive about donating computing time for a Grid or about running applications on a Grid. Survey - Conclusions

RI WP Asia 1-EPIKH School for Application Porting 8 Connecting a large number of resources and make them available for a set of applications Resources belong to different owners Heterogeneous system (HW, SW, network) Typically for Desktop Grids: it is the Desktop Grid computer that initiates all communications: there is no central control over the desktop computer Most computers in a Desktop Grid are desktop computers, but they can also be notebooks, or servers, or mobile phones A Grid definition

RI WP Asia 1-EPIKH School for Application Porting 9 What is Grid? Using physical resources belonging to different organisations for a set of applications Main problem: crossing (administrative) boundaries between the organisations

RI WP Asia 1-EPIKH School for Application Porting 10 Processors - compute Grid Storage - storage Grid Data bases, repositories - data Grid Sensors - sensor Grid Resources that can be shared

RI WP Asia 1-EPIKH School for Application Porting 11 Citizens’ and SMEs’ PC’s: Desktop Grid Thousands or even millions of owners Company divisions: Enterprise Grid A dozen or so organisations within one enterprize Universities and research institutes - eScience Grid Up to a few hundred (in the case of EGEE, a pan-European Grid) Different owners of resources

RI WP Asia 1-EPIKH School for Application Porting 12 Recipe: Take resources Connect them through a fast network Add Grid middleware And your Grid is ready But you have to port your applications to the Grid A Grid is an infrastructure

RI WP Asia 1-EPIKH School for Application Porting 13 Almere is a new town in the Netherlands Good cradle for the world’s first CityGrid Desktop Grid used for scientific applications: sharing CPU power Back-Up Grid experiment to share hard disk space (as part of EU BEinGRID project) Build-on-Grid and Virtual Reality based environment to allow people to build their own house in virtual space with all the right data and checking permits before it is built in reality. (Feasibility study for municipality finished.) An example AlmereGrid - World’s first CityGrid

RI WP Asia 1-EPIKH School for Application Porting 14 Applications on the Grid Science Grid applications Build-on-Grid Own house construction HealthGrid diagnosis agent based AlmereGrid infrastructure e-Education using Grid Strengthen knowledge infra- structure of the city BEinGRID backup/ restore Broadband network For science and beyond

RI WP Asia 1-EPIKH School for Application Porting 15 Agenda Basic techniques Open source examples XtremWebBOINC Commercial examples Commercial examples LSF Desktop DatasynapseParabon Desktop Grids - Introduction

RI WP Asia 1-EPIKH School for Application Porting 16 Resource computers The resource computers (also called workers, clients,..) execute the work User system or interface Interface for the Grid user to submit his/her application, monitor the progress, and retrieve results Grid server Handles the requests from the resource computers to get new work or submit the results Handles the requests from the users to execute jobs Basic structure

RI WP Asia 1-EPIKH School for Application Porting 17 Companies Residential areas Universities & research institutes Grid organisation Basic Desktop Grid (players)

RI WP Asia 1-EPIKH School for Application Porting 18 Grid resources Grid users Grid server Basic Desktop Grid (components)

RI WP Asia 1-EPIKH School for Application Porting 19 Volunteer Desktop Grids Uses a pull-model: it is always the resource computer that asks for work and initiates all communication with the server There is an application repository: users can only run validated trusted applications The resources, however, are untrusted and volatile: you are not sure they deliver the correct answer

RI WP Asia 1-EPIKH School for Application Porting 20 Local Desktop Grids Organisations (universities, companies) can also run an internal Desktop Grid. Can be composed of trusted resources Use spare capacity Can also include servers and clusters

RI WP Asia 1-EPIKH School for Application Porting 21 Desktop Grid applications There is --in general-- no communication between the resources Hence you can only run a subclass of parallel programs: master-worker, parameter sweep or monte-carlo In volunteer computing the network bandwidth and the HDD/CPU capacity can be rather limited But there can be large numbers of resources

RI WP Asia 1-EPIKH School for Application Porting 22 XtremWebXtremWeb Developed by IN2P3 and INRIA in France Written in Java Can be used for all type of Grids, from simple local to advanced volunteer Desktop Grids

RI WP Asia 1-EPIKH School for Application Porting 23 BOINCBOINC Middleware designed for volunteer Desktop grid computing Used for the largest Grid in the world: More than 4 million computers world wide have BOINC installed Many other projects use BOINC

RI WP Asia 1-EPIKH School for Application Porting 24 World Community Grid - IBM ( PCs Leiden Classical Grid - Education on Grid ( PCs SZTAKI - Hungarian Grid ( PCs AlmereGrid ( PCs PS3GRID (Based on Playstations) ( Some other Desktop Grids

RI WP Asia 1-EPIKH School for Application Porting 25 BOINC Structure BOINC is organised around projects: a project is typically one application Resource computers (BOINC clients) can connect to one or more projects A Grid server can run more projects: but most only have one or perhaps a few User interface is integrated with the project

RI WP Asia 1-EPIKH School for Application Porting 26 BOINC installation Installing a BOINC Grid is not click and play Collection of C++ programs, PHP scripts and other sources, with many dependencies on operating system, the database, and other tools Easiest way: get one of the distributions. For instance the SZTAKI BOINC distribution. This installs relatively fast if you start with a Debian 4.0 system

RI WP Asia 1-EPIKH School for Application Porting 27 BOINC Client on resource Customisable, easy-to-install Easy view and complex view with a lot of details on what is happening

RI WP Asia 1-EPIKH School for Application Porting 28 BOINC applications BOINC applications have to be compiled with calls to a special BOINC API. There are some wrappers for executables, like genwrapper (developed by MTA SZTAKI) More extensive API’s exist for instance for master worker-applications: DC-API (developed by MTA SZTAKI)

RI WP Asia 1-EPIKH School for Application Porting 29 Q. A Desktop Grid consists of otherwise unused computing sources. So it is for free? A. It is much cheaper than any other solution, but of course if you use a computer it uses power and produces heat Q. Are Desktop Grids Green? A. Yes: less computers have to be built. Q. So we do not need supercomputers anymore? A. No. Although there are many applications that can run on a Desktop Grid, some require fast communication between the processors. For those supercomputers are more appropriate Q. It took me a long time as a user to get used to Glite within EGEE. Do I have to learn a new Grid technology A. No. The EDGeS project provides a seamless bridge between EGEE and Desktop Grids Desktop Grid QA

RI WP Asia 1-EPIKH School for Application Porting International Desktop Grid Federation Main vehicle for dissemination during the project lifetime and for continuation after the project Federation is open for organisations and individuals in research and industry A member organisation Focal point for DCI/Desktop operators and Application developers Website: discussion fora, events and workshops Also supported by EDGI (For European chapter) 30 RI WP

RI WP Asia 1-EPIKH School for Application Porting 31 RI WP The International Desktop Grid Federation and support from DEGISCO

RI WP Asia 1-EPIKH School for Application Porting 32 RI WP Project work plan WP1 Management WP2 Infrastructure support Technically support the setting up and operation of new desktop grids, especially in ICPC countries. Both local desktop grids, inside organisations, and public volunteer desktop grids will be supported. WP3 Application support The infrastructure that is supported by WP2, will be used by scientific applications that will be run by researchers in ICPC countries or collaborations including researchers from ICPC countries and Europe. WP4. Communication, dissemination, and community management Create awareness amongst all possible stakeholders; recruit new resource providers, new grid operators, new application developers, and new scientific users. It will also disseminate project results and provide training. Create awareness amongst all possible stakeholders; recruit new resource providers, new grid operators, new application developers, and new scientific users. It will also disseminate project results and provide training. 32

RI WP Asia 1-EPIKH School for Application Porting 33 SummarySummary Infrastructures like HPC, cluster grids, desktop grids and cloud have their own roles which are complementary They should go hand in hand and collaborate instead of compete Contact DEGISCO or ISA RAS for more information and collaboration opportunities Join the International Desktop Grid Federation

RI WP Asia 1-EPIKH School for Application Porting QUESTIONS ??? 34

RI WP Asia 1-EPIKH School for Application Porting 35 Commercial Desktop Grids Desktop Grids are a big commercial success Many large companies (pharmaceutical, engineering, banks,..) have installed big Desktop Grids But they do not tell much about it: it is a competitive advantage and proven technology Several companies provide Desktop Grid middleware; although some now call in differently because they do not want to be associated with the Globus and gLite based Grids that by many are seen as failed in the commercial area

RI WP Asia 1-EPIKH School for Application Porting 36 DatasynapseDatasynapse More than 1 million computers run Datasynape software in the world. Making it, probably, the most successful Grid middleware after BOINC Acquired by TIBCO (Cloud provider) recently for US$28 million

RI WP Asia 1-EPIKH School for Application Porting 37 Platform LSF Desktop Platform computing is involved in job management systems for already many years Platform LSF is mainly focused at managing systems with many processors and (large) clusters. This works in push mode Addition: LSF Desktop for Windows machines. This works in pull mode and is used to connect Desktops to an LSF cluster

RI WP Asia 1-EPIKH School for Application Porting 38 Frontier Grid platform Works as a traditional Desktop Grid Fancy, easy to use user GUI ParabonParabon

RI WP Asia 1-EPIKH School for Application Porting 39 And there are more... Fura: Grid systems, Spain ProActive: ActiveEon, France Digipede, USA, …