FESR Consorzio COMETA - Progetto PI2S2 Introduction to Grid Computing Pietro Di Primo INFN – Catania , Catania

Catania, Fundamentals of Grid Computing

Catania, The Grid and the Internet It is considered the next step in the evolution of distributed computing Adopts the Internet as communication infrastructure for exchanging information It is considered as the equivalent of the Internet in terms of exchanging of computational power

Catania, The Grid idea The user: –Does not need to know what stays beyond the user interface –Can access to a massive amounts of computational power through a simple terminal The society: –Can extend grid facilities at any moment –Manages the architecture of the grid –Defines policies and rules for accessing to grid resources Some computing farms produce computing power to be shared Computing power is made available over the Internet One user wants to access to intensive computational power He/she comes to an agreement with some society that offers grid services Now the user accesses to grid facilities as a grid user The society will provide for grid facilities allowing the user to access to its grid resources and providing for proper tools

Catania, Why Computing Grids now? Because the amount of computational power needed by many applications is getting very huge Because the amount of data requires massive and complex distributed storage systems To make easier the cooperation of people and resources belonging to different organizations To access to particular instrumentation that is not easily reachable in a different way Because it is the next step in the evolution of distribution of computation Thousands of CPUs working at the same time on the same task From hundreds of Gigabytes to Petabytes (10 15 ) produced by the same application. People of several organizations working together to achieve a common goal Because it cannot be moved or replicated or its cost is too much expensive. To create a marketplace of computational power and storage over the Internet

Catania, Who is interested in Grids? Research community, to carry out important results from experiments that involve many and many people and massive amounts of resources Enterprises that can have huge computation without the need for extending their current informatic infrastructures Businesses, which can provide for computational power and data storage against a contract or for rental

Catania, Virtual Organizations Virtual Organization (VO) –Is a collection of people and resources working together to achieve the same goal –It is cross-domain (people and resources) One user –Identified by his/her personal X.509 certificate issued by trusted Certification Authorities (CA) –Can belong to more than one VO at the same time –Does not require detailed knowledge of grid technologies to access to the Grid

Catania, Virtual Laboratory A new way of cooperating in experiments A platform that allow scientists to work together in the same “Virtual” Laboratory Strictly correlated to Grids and Virtual Organizations

Catania, Properties of Grids Transparency –The complexity of the Grid architecture is hidden to the final user –The user has to be able to use a Grid as it was a unique virtual supercomputer –Resources must be accessible setting their location apart Openness –Each subcomponent of the Grid is accessible independently of the other components Heterogeneity –Grids are composed by several and different resources Scalability –Resources can be added and removed from the Grid dynamically Fault Tolerance –Grids must be able to work even if a component fails or a system crashes Concurrency –Different processes on different nodes must be able to work at the same time accessing to the same resources

Catania, Challenged Issues in Grids (i) Security –Authentication and authorization of users –Confidentiality and not repudiation Information Services –Discovery and monitoring of Grid resources –Checking for health-status of resources –To be the base for any decision making processes File Management –Creation, modification and deletion of files –Replication of files to improve access performances –Ability to access to files without the need to move them locally to the code Administration –Systems to administer Grid resource respecting local administration policies

Catania, Challenged Issues in Grids (ii) Resource Brokering –To schedule tasks across different resources –To make optimal or suboptimal decisions –To reserve (in the future) resources and network bandwidth Naming services –To name resources in an unambiguous way in the Grid scope Friendly User Interfaces –Because most of Grid users have nothing to do with computing science (physicians, chemistries, videogame players...) –Graphical User Interfaces (GUIs) –Grid Portals (very similar to classical Web Portals) –Command Line Interfaces (CLIs) for experts

Catania, Application Areas (1/2) Physicical Science Applications –GryPhiN, –Particle Physics DataGrid (PPDG), –GridPP, –AstroGrid, Life Science Applications –Protein Data Bank (PDB), –Biomedical Informatics Research Network (BIRN), –Telemicroscopy, –myGrid,

Catania, Application Areas (2/2) Engineering Oriented Applications –NASA Information Power Grid (IPG), –Grid Enabled Optimization and Design Search for Engineering (GEODISE), Commercial Applications –Butterfly Grid, –Everquest, E-Utility –ClimatePrediction experiment,

Catania, The Architecture and the Middleware

Catania, Architecture Overview

Catania, The Single Sign On (SSO) –It is a way to sign in to the Grid just one time and access to all services you need –It is done via proper command lines of the User Interface and providing a valid certiicate –A user submits his certificate to the VOMS server VO Membership Service (VOMS) –Issues proxies to users with VOMS extensions –Assigns groups/roles/capabilities to the users –Verifies user credentials (ceritificate)

Catania, Computing Element (CE) Worker Nodes –More than one (farm) Local Resource Manager (LRMS) –It is NOT a Grid component –Local administration –Dispatches job across Worker Nodes according to local policies –Accessed by local users Computing Element –Service oriented –Exports computing power of the LRMS to grid users –Maps grid users to local users –Wraps a “grid job” into a “local job” Gatekeeper –Access point of the CE –Manages one or more job queues –Transparent interface to the LRMS To execute grid jobs, a specific environment must be set

Catania, Storage Element (SE) Mass Storage System (MSS) –It is NOT a grid component –Manages the local storage device –Hard disks, digital tapes Storage Resource Manager (SRM) –Is the grid compliant interface to access to the storage system –Maps grid users to local users –Access and control to the MSS Grid FTP compliant daemons –Grid compliant daemons for uploading/downloading of grid files –On top of the local MSS –Protocols: gsiftp, rfio, dcap Grid client –Any user/process with a valid proxy

Catania, Data Management LHC File Catalogue (LFC) –Hierarchical file catalogue –Each VO has its own catalogue Logical File Name (LFN) –The name of a file within the LFC –To each LFN correspond one or more replicas stored on different SEs –lfn://grid/cometa/file.txt Storage URL (SURL) –The name of a replica within a SE –To each SURL correspond more TURLs –srm://se1.cometa.it/some/path/file.txt Transport URL (TURL) –Is the name of the physical file plus the grid FTP protocol to access to it –gsiftp://se1_disk2.cometa.it/local/path/f ile1.txt

Catania, Information System Globus Meta Directory Server (MDS) –Hierarchical, based on LDAP servers –The user can query the system at every level –The higher the information is in the tree, the older it is Grid Resource Information Service (GRIS) –Collects info on that resource –Static or dynamic info –Uses sensors Grid Index Information Service (GIIS) –One for each site –Collects info from below GRISes –Caches info according to its validity time Berkeley Database Information Index (BDII)

Catania, Workload Management System (1/4) The grid user describes a job via a Job Description Language (JDL) document. Some input files (Input Sandbox) can be attached to the JDL doc. The grid user submits the JDL job using the command line interface and waits for reply. The Resource Broker gets and stores the JDL document together with attached input files. The just generated jobId is sent back to the user to refer to that job univocally in the future

Catania, Workload Management System (2/4) The Resources Broker executes a special algorithm (matchmaking) and chooses a Computing Element according to best-fit rules The job is handed to the chosen CE together with the Input SandBox The Computing Element accepts the job and queues it. The job starts execution over the Local Resources Management System (LRMS)

Catania, Workload Management System (3/4) When the job terminates, the produced output is sent back to the Resource Broker The Resource Broker gets the results and the Output Sandbox and stores them in the local repository At the same time, the Computing Element notifies the Logging & Bookkeeping Now the job output is available on the Resource Broker

Catania, Workload Management System (4/4) The User queries the Logging and Bookkeping to have a look on his/her jobs and realizes that the job has terminated. The User downloads the Output Sandbox from the Resource Broker. The Resource Broker clears all no more needed info from its repository to save space. The job life cycle has terminated !!!

Catania, References Consorzio COMETA Homepage – INFN Production Grid – gLite homepage – gLite Architecture Document –

Catania, Questions…