Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 0 Lezione 2a - 14 ottobre 2009 Il materiale didattico usato in questo corso è stato mutuato da quello utilizzato da Paolo Veronesi per il corso di Griglie Computazionali per la Laurea Specialistica in Informatica tenuto nellanno accademico 2008/09 presso lUniversità degli Studi di Ferrara. Paolo Veronesi Università degli Studi di Bari – Corso di Laurea Specialistica in Informatica Tecnologia dei Servizi Grid e cloud computing A.A. 2009/2010 Giorgio Pietro Maggi

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 1 Grid Architecture

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 2 Grid Architecture: Definition (1/2) The Grid Architecture: identifies the fundamental system components (services and VOs); specifies purpose and function of these components; indicates how these components interact with each other. Functions: It identifies key areas in which services are required; It defines standard protocols and APIs to facilitate creation of interoperable Grid systems and portable applications. Grid architecture protocols define the basic communication mechanisms by which: Virtual Organization users and resources negotiate, establish, manage and exploit sharing relationships; different services interact. Grid architecture services are the components whose standardization facilitates extensibility, interoperability, portability and code sharing.

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 3 Grid Architecture: Definition (2/2) S1 S2 S4 S3 U3 U5 U8 U7 U6 U1 U2 VO1 VO2 COMPONENTS 1. VOs and their users 2. Services 3. Protocols: - between services and VOs protocol

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 4 Resource Sharing Address security and policy concerns of resource owners and users Are flexible enough to deal with many resource types and sharing modalities Scale: to large number of resources, many participants, many program components to large data management tasks and computing tasks

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 5 Service Sharing 1) Need for interoperability when different groups want to share resources Diverse components, policies, mechanisms E.g., standard notions of identity, means of communication, resource descriptions 2) Need for shared infrastructure services to avoid repeated development, installation E.g., one port/service/protocol for remote access to computing, not one per tool and/or application E.g., use of shared Certificate Authorities as they are expensive to run A common need for protocols & services

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 6 Layered Grid Architecture (by Analogy to the Internet Architecture) Application Fabric Controlling elements locally: Access to, & control of, resources Connectivity Talking to Grid elements: communication (Internet protocols) & security Resource Sharing single resources: negotiating access, controlling use Collective Coordinating multiple resources: ubiquitous infrastructure services, app-specific distributed services Internet Transport Application Link Internet Protocol Architecture Grid Architecture Internet Architecture

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 7 Layered Grid Architecture Layers are ment to group services of similar nature in the same class They do not intend to prescribe how services at different layers interact. Communication between services at the same and/or different layer is possible, depending on the scenario. Example 1: The workload management service (Collective layer) uses the Resoure discovery service (still Collective layer) to find resources that match the users rquirements Example 2: an application may submit a job: By invoking the Workload manager service (re-submission, logging and bookkeping, input/output management, Collective) By invoking the resource layer through a standardized API (Resource) By submitting directly to a specific local resource instance (Fabric)

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 8 Protocols, Services, and APIs Occur at Each Level Languages/Frameworks Fabric Layer Applications Local Access APIs and Protocols Collective Service APIs and SDKs Collective Services Collective Service Protocols Resource APIs and SDKs Resource Services Resource Service Protocols Connectivity APIs Connectivity Protocols

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 9 Fabric Layer (1/2) Examples : Computing farm Mass storage device file catalogs network link file systems sensors, etc. It provides access to the resources to which shared access is mediated by Grid protocols. Fabric components implement local, resource-specific operations (through internal protocols), which are transparent to the upper layers thanks to connectivity and resource level protocols, that define interfaces, not the physical internal characteristics. Operations at the fabric layer are triggered by resource sharing operations at a higher level

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 10 Fabric Layer: Capabilities (2/2) Examples of capabilities Computational resource: Start programs Monitor execution and corresponding processes Management of resources (e.g. advance reservation) Enquiry of hw/sw capabilities, current load, queue waiting time, etc. Storage resource: Put/get files Third party and high-performance data transfer Read/write subset of file Management of disk space, disk bandwidth, network bandwidth, CPU, etc. Enquiry of hw/sw characteristics, available space, bandwidth utilziation, etc.

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 11 Connectivity Layer: Protocols & Services It supports secure communication between Fabric-layer resources by defining the core communication and authentication protocols required for grid-specific network functions. Communication: transport (e.g. IP), naming (e.g. DNS), routing, etc. (the TCP/IP protocol stack) Security: the Grid Security Infrastructure (GSI) Uniform authentication, authorization, and message protection mechanisms in multi-institutional setting Single sign-on: users must be able to authenticate just once and then have access to multiple resources, without further user intervention; Delegation: a users program can access the resources on which the program owner is authorized. Integration with local security solutions: Grid security needs to interoperate with local security solutions adopted by the resource managers User-based trust relationship: a user can access to resources in different domains without requiring security administrators in each domain to interact with others in different domains. In other words, authorization is only based on the users certificate, not on the authentication/authorization performed by other servers in different domains. Public key cryptography: Secure Socket Layer (SSL) Supporting infrastructure: Certificate Authorities, certificate & key management

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 12 Resources Layer: Protocols & Services Resource layer defines protocols, APIs, and SDKs for: secure negotiations, initiation, monitoring control, accounting, and payment of sharing operations on individual resources. Resource layer implementation relies on the Fabric layer functions. Two main components define this layer: information protocols: used to obtain the information about the structure and state of the resource, e.g.: configuration, current load and usage policy. management protocols: used to negotiate access to the shared resource, specifying for example QoS, advanced reservation, etc. Examples: Access to compute cluster, storage, network information Invocation of resource service providers Access to local scheduler

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 13 Collective Layer: Protocols & Services The Collective Layer contains protocols and services that coordinate multiple resources concurrently, i.e. they capture interactions among a collection of resources. It supports a variety of sharing behaviors without placing new requirements on the resources being shared. Examples: Directory services: they allow VO participants to discover the existence and/or properties of VO resources. Co-allocation, scheduling: they allow VO participants to request the allocation of one or more resource for a specific purpose and the scheduling of tasks on the appropriate resources. Monitoring: support the monitoring of VO resources (intrusion, failure, load,..) Data replication: supports the management of VO storage resources to maximize data access performance with respect to metrics such as response time, reliability, etc. Workload management: description, use and management of complex workflows (e.g. submission of jobs with mutual dependencies) Authorization servers, accounting, collaboratory services (synchronous/asynchronous exchange of messages)

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 14 Application Layer It includes user applications that operate within a Virtual Organization environment. Applications are constructed such that invocation of services and use of protocols defined at any layer are possible. For example, a given user can submit jobs by invoking services at different layers: Collective Layer: by submitting to a Workload Manager (it can provide: internal queue in: case of submission failure, periodic re-submission, access to logging and bookkeeping information, input/output file management) Resource Layer: by submitting directly to a compute resource scheduler (e.g. a local cluster) Connectivity Layer: by submitting to a remote cluster Fabric Layer: by submitting to a local cluster

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 15 Three Generations of Grid Source: Charlie Catlett Standardization is key for third-generation grids! Local metacomputers –Distributed file systems –Site-wide single sign-on "Metacenters" explore inter-organizational integration Totally custom-made, top-to-bottom: proofs of concept Local metacomputers –Distributed file systems –Site-wide single sign-on "Metacenters" explore inter-organizational integration Totally custom-made, top-to-bottom: proofs of concept Utilize software services and communications protocols developed by grid projects: –Condor, Globus, UNICORE, Legion, etc. Need significant customization to deliver complete solution Interoperability is still very difficult! Utilize software services and communications protocols developed by grid projects: –Condor, Globus, UNICORE, Legion, etc. Need significant customization to deliver complete solution Interoperability is still very difficult! Common interface specifications support interoperability of discrete, independently developed services Competition and interoperability among applications, toolkits, and implementations of key services Common interface specifications support interoperability of discrete, independently developed services Competition and interoperability among applications, toolkits, and implementations of key services We are here!

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 16 Who Defines Standard for Grid OGF: Open Grid Forum community of users, developers, and vendors leading the global standardization effort for Grid computing OASIS: Organization for the Advancement of Structured Information Standards not-for-profit consortium that drives the development, convergence and adoption of open standards for the global information society DMTF: Distributed Management Task Force industry organization leading the development, adoption and promotion of interoperable management standards and initiatives W3C: World Wide Web develops interoperable technologies (specifications, guidelines, software, and tools) to lead the Web to its full potential

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 17 Open Grid Forum Birthed in high performance computing community in 1998 Merged with European & Asian grid efforts in groups and 81 documents Birthed in enterprise data center community in 2004 Raised awareness of grids in end-user organizations 5 groups and several important documents Merger completed, June 06; OGF Launched September of 06

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 18 Broad and Narrow Grids OGF distinguishes between broad and narrow Grids A Broad Grid is any collection of distributed services Marketing-oriented, not rigorous definition A Narrow Grid Collaboration Grid Data Center Grid Cluster Grid

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 19 Collaboration Grids Multiple institutions, secure, widely distributed, VOs Collaborative agreements & commercial partnerships Financial Model: Increase overall revenue Examples: EGEE

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 20 Data Center Grid Centralized management of multiple platforms Aggregation of enterprise resources and applications Financial Model: Reduce Total Cost Ownership (TCO) Examples: Google, eBay, Amazon

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 21 Cluster Grid Networks of Workstations, Blades, etc. Cycle scavenging, Homogeneous workload Financial Model: Lower marginal costs

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 22 How Are Grids Used? High-performance computing Collaborative data-sharing Collaborative design Drug discovery Financial modeling Data center automation High-energy physics Life sciences E-Business E-Science

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 23 OGSA The Open Grid Services Architecture (OGSA) architecture describes the functionality required to implement a Grid infrastructure, and the rendering of this functionality into service definitions

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 24 OGSA Capabilities OGSA OGSA profiles Web services foundation

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 25 OGSA Capabilities Execution Management Job description & submission Scheduling Resource provisioning Execution Management Job description & submission Scheduling Resource provisioning OGSA OGSA profiles Web services foundation

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 26 OGSA Capabilities Execution Management Job description & submission Scheduling Resource provisioning Execution Management Job description & submission Scheduling Resource provisioning Data Services Common access facilities Efficient & reliable transport Replication services Data Services Common access facilities Efficient & reliable transport Replication services OGSA OGSA profiles Web services foundation

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 27 OGSA Capabilities Execution Management Job description & submission Scheduling Resource provisioning Execution Management Job description & submission Scheduling Resource provisioning Data Services Common access facilities Efficient & reliable transport Replication services Data Services Common access facilities Efficient & reliable transport Replication services Resource Management Discovery Monitoring Control Resource Management Discovery Monitoring Control OGSA OGSA profiles Web services foundation

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 28 OGSA Capabilities Execution Management Job description & submission Scheduling Resource provisioning Execution Management Job description & submission Scheduling Resource provisioning Data Services Common access facilities Efficient & reliable transport Replication services Data Services Common access facilities Efficient & reliable transport Replication services Self-Management Self-configuration Self-optimization Self-healing Self-Management Self-configuration Self-optimization Self-healing Resource Management Discovery Monitoring Control Resource Management Discovery Monitoring Control OGSA OGSA profiles Web services foundation

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 29 OGSA Capabilities Information Services Registry Notification Logging/auditing Information Services Registry Notification Logging/auditing Execution Management Job description & submission Scheduling Resource provisioning Execution Management Job description & submission Scheduling Resource provisioning Data Services Common access facilities Efficient & reliable transport Replication services Data Services Common access facilities Efficient & reliable transport Replication services Self-Management Self-configuration Self-optimization Self-healing Self-Management Self-configuration Self-optimization Self-healing Resource Management Discovery Monitoring Control Resource Management Discovery Monitoring Control OGSA OGSA profiles Web services foundation

Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 30 OGSA Capabilities Security Cross-organizational users Trust nobody Authorized access only Security Cross-organizational users Trust nobody Authorized access only Information Services Registry Notification Logging/auditing Information Services Registry Notification Logging/auditing Execution Management Job description & submission Scheduling Resource provisioning Execution Management Job description & submission Scheduling Resource provisioning Data Services Common access facilities Efficient & reliable transport Replication services Data Services Common access facilities Efficient & reliable transport Replication services Self-Management Self-configuration Self-optimization Self-healing Self-Management Self-configuration Self-optimization Self-healing Resource Management Discovery Monitoring Control Resource Management Discovery Monitoring Control OGSA OGSA profiles Web services foundation