Linux on zSeries Module 9: Grid

Objectives State how grid computing differs from regular distributed computing. State the “grid problem”. Define a VO. List three things a VO must take into consideration when setting up relationships for its members. State some reasons why using protocols in a grid is a necessity. Briefly describe the grid fabric layer. Briefly describe the grid connectivity layer and its functions. Briefly describe the grid resource layer and its functions. Briefly describe the grid collective layer and its functions. Briefly describe the grid applications layer and its functions. List the two technologies that merged in order to create grid services.

Objectives Describe how using WSDL benefits the OGSA model. Describe how using the Globus tool kit benefits the OGSA package. List a few assumptions of the OGSA service model. State the role of hosting environments with the OGSA model. List three possible VO structures that can be set up using OGSA. List four characteristics of an on demand business. State some advantages to using open standards in a grid environment. List four features of zSeries® that make it an optimal platform for on demand scenarios. Define non-disruptive growth and state how zSeries provides this. Define Virtual Server Services. Define on/off capacity on demand.

Grid Computing “Grid” computing differs from regular distributed computing due to its focus on large-scale resource sharing, innovative applications, and high-performance orientation. Shares resources among virtual organizations (VOs). This sharing between organizations creates unique authentication, authorization, resource access, resource discovery, and other technical challenges due to the need to achieve various qualities of service when running on top of different native platforms. Grid Architecture deals with these challenges through protocols, services, application programming interfaces (APIs), and software development kits (SDKs), that are categorized according to their roles in enabling resource sharing, accelerating code development, enable code sharing, and enhance application portability.

Virtual Organizations Are, “dynamic collections of individuals, institutions, and resources that need flexible, secure, coordinated resource sharing to achieve a common goal.” These resources are not simple file exchanges, but rather direct access to computers, software, data, and other resources that may be necessary for a range of collaborative problem-solving and resource brokering strategies emerging in industry, science, and engineering. An elaborate set rules, establishing who can do what with whom, is needed. Examples of VOs are: Application service providers, storage service providers, cycle providers, consultants engaged by a car manufacturer to perform scenario evaluation during planning for a new factory, members of an industrial consortium bidding on a new aircraft, and a crisis management team and the databases and simulation systems, that they use to plan a response to an emergency situation.

The Grid Problem Resource sharing and coordinated problem solving in dynamic, multi-institutional virtual organizations.

The Grid Problem The grid problem: Need for unique coordinated resource sharing. Need for problem solving in dynamic, multi-institutional virtual organizations. Things a VO needs to take into consideration in order to survive as a successful entity: Highly flexible sharing relationships, ranging from client-server to peer-to-peer, which allows for sophisticated and precise levels of control over how shared resources are used. Fine-grained and multi-stakeholder access control, delegation, and application of local and global policies; for sharing of varied resources, ranging from programs, files, and data to computers, sensors, and networks. Diverse usage modes, ranging from single-user to multi-user and from performance-sensitive to cost-sensitive; therefore, embracing issues of quality of service, scheduling, co-allocation, and accounting.

Grid Complements Distributed Grid complements existing distributed computing technologies. Resource sharing is conditional, each resource owner makes resources available; subject to constraints on when, where, and what can be done. This in turn makes authentication and authorization necessary. Sharing relationships can vary dynamically over time, in terms of the resources involved, the nature of the access permitted, and the participants to whom access is permitted.

Why is Grid Possible Today The Internet as an infrastructure. increasing bandwidth, advanced services Advances in storage capacity. terabytes, petabytes per site Increased availability of compute resources. clusters, supercomputers, and so on Advanced applications. Simulation-based design, advanced scientific instruments, and so on Protocol and standards such as those supported by Globus. Web service architecture.

Grid Architecture Interoperability, sharing among any potential participants, is key. Grid facilitates this by being a protocol-based architecture, with protocols defining the basic mechanisms by which VO users and resources negotiate, establish, manage, and exploit sharing relationships. By having this strong set of standards, Grid facilitates extensibility, interoperability, portability, and code sharing. Protocols also make it easier to define standard services that provide enhanced capabilities. Globus Toolkit is an open source reference implementation of key Grid protocols that supports a wide variety of major e-science projects.

Information Infrastructure of Today Network-centric: simple, fixed-end systems; few embedded capabilities; few services; no user-level quality of service

Infrastructure of Tomorrow: Not Just “Faster and More Reliable” Caching Resource Discovery QoS Application-centric: heterogeneous, mobile end-systems; many embedded capabilities; rich services; user-level quality of service

Grid Services Architecture High-energy physics data analysis Collaborative engineering On-line instrumentation Applications Regional climate studies Parameter studies Application Toolkit Layer Distributed computing Data- intensive Collab. design … Remote control Grid Services Layer Information Resource mgmt Security Data access Fault detection . . . Grid Fabric Layer Transport Multicast Instrumentation Control interfaces QoS mechanisms . . .

Grid Services (“Middleware”) Standard services that: Provide uniform, high-level access to a wide range of resources (including networks). Address interdomain issues of security, policy, and so on. Permit application-level management and monitoring of end-to-end performance. Middleware-level and higher-level APIs and tools targeted for application programmers: Map between application and Grid.

Grid Services (Open Grid Services Architecture: OGSA ) Dynamic and volatile – Requested can be switched in-and-out dependant on machines that are being most utilized. aAd hoc – No central location and no central control. Large – Orchestrates hundreds of machines and requests at any one given time. Long lived – Simulations can take weeks, such as those being done for protein synthesis and DNA reasearch. Grid Services - Require information about functionality, availability, and interfaces of the various services. Service discovery and brokering uses metadata descriptions. Service composition is controlled and supported by metadata descriptions.

Grid Services Grid technologies can be aligned with the Web services architecture to capitalize on desirable Web services properties. This combination of Grid and Web service technologies become realized in the form of an Open Grid Services Architecture (OGSA). OGSA uses the Web Services Description Language (WSDL) to achieve self-describing, discoverable services and interoperable protocols, with extensions to support multiple coordinated interfaces and change management. OGSA utilizes the advantages of the Globus Toolkit to define conventions and WSDL interfaces for a Grid service, a service instance supporting reliable and secure invocation, lifetime management, notification, policy management, credential management, and virtualization. OGSA also defines interfaces for the discovery of Grid service instances and for the creation of transient Grid service instances. This combination results in a standards-based distributed service system that supports the creation of the sophisticated distributed services, required in modern enterprise and inter-organizational computing environments.

How zSeries Fits In With Grid Linux on zSeries is currently being used to support research in Grid computing such as at the University of Florida. This research shows Grid computing also plays a role in business applications. For example, a financial institution based in New York might set up an Internet-based grid so that their branch offices in Europe could use the company computer resources in New York. This would save the company the expense of purchasing and maintaining hardware and software on both continents. The z800 contributed for this grid research is being used so the Grid may utilize virtualization technology at the machine, network, data, and application levels to dynamically create virtual information grids per user or per application. The intended users of this Grid computing approach include worldwide communities of scientists, engineers in nanotechnology, and computer science. The virtualization capability available on the zSeries allows the mainframe to be shared by multiple researchers, each with separate and distinct applications on a single application on a single piece of hardware and simulate a real grid. Virtualization provides security on two levels, useful for grid experimentation: By preventing the user from interacting with the underlying hardware, it protects the hardware from being effected by any vulnerability due to security holes of the user application. It also protects users from accessing the same physical resources because they are on completely decoupled virtual machines.

How Linux Fits With Grid Grid computing is dedicated to creating a system that was open source on a common platform. Linux implementation on the zSeries is important to Grid research, which has thus far been almost entirely on Unix Linux advantages: It is open source and its freely available. Scientific applications developed over the past 30 years in Unix are already making the transition (such as, ported) to Linux, including the middleware needed to run Grid. Reliability and ease a Linux system can be managed are additional pluses for using the operating system in the Grid environment. Linux helps support the OGSA Architecture. Linux makes it easy for a university and scientific community to manage limited professional IT resources and a support staff that is often made up of primarily students.

Grid Architecture Diagram

Fabric Layer: Interfaces to Local Control Provides the resources that Grid protocols mediate access to. Fabric components implement the local, resource-specific operations that occur on specific resources (whether physical or logical) as a result of sharing operations at higher levels. There is a strong link between the functions implemented at the Fabric level and the sharing operations supported. For example: Computational resources require mechanisms for starting programs and for monitoring and controlling the execution of the resulting processes. Storage resources require mechanisms for putting and getting files. Network resources require management mechanisms that provide control over the resources allocated to network transfers (such as prioritization and reservation). Code repositories require mechanisms for managing versioned source and object code. Catalogs require mechanisms for implementing catalog query and update operations.

Connectivity Layer: Communication Defines core communication and authentication protocols required for Grid-specific network transactions. Communication protocols enable the exchange of data between Fabric layer resources. Authentication protocols build on communication services to provide cryptographically secure mechanisms for verifying the identity of users and resources. Communication requirements include transport, routing, and naming (protocols will probably be taken from the TCP/IP stack). Security requirements also necessitate that any solutions be based on existing standards (again probably form the Internet protocol suite) whenever possible. VO authentication solutions should include: single sign on delegation integration with various local security solutions user-based trust relationships

Resource Layer: Sharing Single Resources Builds on Connectivity layer communication and authentication protocols to define protocols (and APIs and SDKs) for the secure negotiation, initiation, monitoring, control, accounting, and payment of sharing operations on individual resources. Resource layer implementations of these protocols call Fabric layer functions to access and control local resources. Resource layer protocols are concerned entirely with individual resources. There are two main types of Resource layer protocols: Information protocols, which are used to obtain information about the structure and state of a resource (for example, its configuration, current load, and usage policy). Management protocols, which are used to negotiate sharing relationships, (such as defining resource requirements). Management protocols serve as a “policy application point,” making sure that the requested protocol operations are consistent with the policy under which the resource is to be shared.

Collective Layer: Coordinating Multiple Resources Contains protocols and services (and APIs and SDKs) that are not associated with any one specific resource, but rather are global in nature and capture interactions across collections of resources. Collective components can implement a wide variety of sharing behaviors without placing new requirements on the resources being shared. Collective layer protocols span the spectrum from general purpose to highly application or domain specific; for example: directory services co-allocation, scheduling, and brokering services monitoring and diagnostics services data replication services grid-enabled programming systems workload management systems and collaboration software discovery services community authorization servers Collective functions can be implemented as persistent services (rather than transient), with associated protocols, or as SDKs (with associated APIs) designed to be linked with applications.

Applications Layer Is composed of the user applications that operate within a VO environment. Applications are constructed in terms of, and by calling upon, services defined at any layer. At each layer there are well-defined protocols that provide access to some useful service, such as resource management, data access, resource discovery, and so on. At each layer, APIs may also be defined as implementation exchange protocol messages with the appropriate service to perform desired actions.

Application Programmer’s View of Grid

Grid Standardization Continuing decentralization and distribution of software, hardware, and human resources makes it essential to achieve and agree upon set of quality of service (QoS) agreements. These can be measured in various different ways such as in terms of common security semantics, distributed workflow, and resource management performance, coordinated fail-over, problem determination services, or other metrics on resources assembled dynamically from enterprise systems, service provider systems, and customer systems. New abstractions and concepts that allow applications to access and share resources and services across distributed, wide area networks are required. The issue is addressed through the nature of the services that respond to protocol messages. Grid is an extensible set of Grid services that may be aggregated in various ways to meet the needs of VOs, which themselves can be defined in part by the services that they operate and share.

The Globus Toolkit Globus Toolkit [25, 29] is a community-based, open-architecture, open-source set of services and software libraries that support Grids and Grid applications. It addresses issues of security, information discovery, resource management, data management, communication, fault detection, and portability. Most relevant to OGSA are: Grid Resource Allocation and Management (GRAM) protocol and the “gatekeeper” service that provides for secure, reliable, service creation, and management of arbitrary computations. Meta Directory Service (MDS), which provides for information discovery through data modeling and a local registry. Grid Security Infrastructure (GSI), which supports single sign on, delegation, and credential mapping to remote computations. Together these are the essential elements of a service-oriented architecture.

Web Services Web services describe a distributed computing paradigm that focus on simple, Internet-based standards (such as eXtensible Markup Language: XML) to address heterogeneous distributed computing. Web services define a technique for describing software components to be accessed, methods for accessing these components, and discovery methods that enable the identification of relevant service providers. Web services standards are being defined within the W3C and other standard bodies and form the basis for major new industry initiatives such as Microsoft (.NET), IBM® (Dynamic on demand Business), and Sun (Sun ONE). One important protocol, the Simple Object Access Protocol (SOAP) provides a means of messaging between a service provider and a service requestor and works independent of the underlying transport protocol (HTTP, FTP, and so on). Another language is the Web Services Description Language (WSDL), which fulfills the need to support the dynamic discovery and composition of services in heterogeneous environments. This is achieved by providing a standard mechanism for defining interface definitions separately from their embodiment within a particular binding. WS-Inspection language is comprised of simple XML and related conventions for locating service descriptions published by a service provider.

OGSA Service Model The basic premise of OGSA is that everything is represented by a service. A network enabled entity that provides some capability through the exchange of messages. Services include; for example, computational resources, storage resources, networks, programs, databases. This uniform service-oriented model means that all components of the environment are virtual. OGSA represents everything as a Grid service: a Web service that conforms to a set of conventions and supports standard interfaces for such purposes as lifetime management. This core set of consistent interfaces, from which all Grid services are implemented, facilitates the construction of hierarchal, higher-order services that can be treated in a uniform way across layers of abstraction.

Services Services within a complex distributed system must be independently upgradeable. Services (and the hosting environments in which they run) must be upgradeable without disrupting the operation of clients. To ensure both these are met, OGSA defines conventions for identifying when a service changes and the degree of backward compatibility, as well as providing a mechanism for refreshing a client’s knowledge of the service. Services include protocol binding for communicating with the service that contain authentication and reliable service invocation. Authentication mechanisms allow the identity of individuals and services to be established for policy enforcement. Reliable service invocation guarantees if a message has been received either once or not at all.

The Role of Hosting Environments While OGSA defines the semantics of a Grid service instance (for example, how it is created, how it is named, how its lifetime is determined, how to communicate with it) it doesn’t place requirements on what a service does or how it performs that service. Grid services are contained within an execution or hosting environment that defines the implementation programming model, programming language, development tools, and debugging tools. Hosting environment also describes how an implementation of a Grid service will meet its obligations with respect to Grid service semantics. Standard grid semantics may be part of the service or provided as a linked library. A hosting environment may be a native operating system environment, or more sophisticated container or component-based environments like J2EE that offer superior programmability, manageability, and flexibility for on demand business.

Using OGSA to Build VO Structures Applications and users need to be able to create services, and to discover and determine the properties of those available. This is provided by OGSA interfaces. Basically, a registry service that logs the services containing sufficient service data for discovery, defines the service set associated with a VO. A Simple hosting environment is a set of resources located within a single administrative domain and supporting native facilities for service management. For example, a J2EE application server, Microsoft .NET system, or Linux cluster. In this setup the environment will be set up as a registry.

Using OGSA to Build VO Structures continued A Virtual hosting environment is a more complex environment in which the sources associated with a VO will span heterogeneous, geographically distributed hosting environments. The environment; for example, can correspond to a set of resources associated with a B2B (Business to Business) partnership. Resources can still be made accessible to a client via exactly the same interfaces as were used for a simple environment through the creation of multiple levels of registries (this process is transparent to the user who believes he/she is only dealing with one registry). Services maintained by the VO can still have unique names and resource management policies can be defined and enforced on the platforms hosting VO services, targeting the VO by this unique name. Collective operations are virtual hosting environments that provides VO participants with more sophisticated, virtual, “collective” or “end-to-end” services. Here the registry keeps track of and advertises factories that create higher-level service instances. Such instances are implemented by asking lower-level factories to create multiple service instances and by composing the behaviors of those multiple lower-level service instances into that single, higher-level service instance.

Commercial Applications of Grid Commercial settings require seamless integration with existing resources and applications, and with tools for workload, resource, security, network QoS, and availability management. Support from OGSA for the discovery of service properties, facilitates the mapping or adaptation of higher-level Grid service functions to such native platform facilities. Service orientation from OGSA also allows us to virtualize resources at multiple levels, allowing the same mechanisms to be used both within distributed Grids supporting collaboration across organizational domains and within hosting environments spanning multiple tiers within a single IT domain. A common infrastructure means that differences (such as visibility and accessibility) derive from policy controls associated with resource ownership, privacy, and security, rather than interaction mechanisms. As enterprise systems are transformed from separate computing resource islands to integrated, multi tiered distributed systems, service components can be integrated dynamically and flexibly, both within and across various organizational boundaries. Just as the Internet began for scientific purposes and spread to the business arena, Grid can follow the same path and provide advantages for partners in B2B.

On Demand Business Characteristics of an on demand business: Responsive Information is immediately available allowing an enterprise to adjust dynamically to any challenge, customer demand, or a market opportunity before it becomes a threat. Focused Can focus on its core competencies and the things that make it stand out from its competitors. Resilient The operating environment is self-managing and regulating. Variable It can choose the best medium between large, upfront investments in IT and simply incurring the variable costs of using the IT resources of a service provider.

On Demand & Open Standards Open standards, like the Grid protocol, need to be used to facilitate a varied infrastructure as possible. Integration and standards allow virtualization of resources that frees users from having to understand how components work. Open standards permit the formation of computing grids that allow an enterprise to share computing resources, thus optimizing the installed base of information technology. Open standards facilitate communication among systems.

On Demand and Linux Open standards facilitate innovation. Open source means more minds and resources working on any given project. Linux is the symbol of collaboration and innovation making it the perfect candidate for integrated environments, especially when combined with high availability and security of zSeries.

zSeries and Business On Demand Business on demand requires an infrastructure that provides optimal performance, real-time responsiveness, application flexibility, and power, and is still easy to use and maintain. Enterprises must also provide a secure and reliable environment for distributed transactions flowing across a collection of dissimilar servers, must deliver continuous availability as seen by end-users, and must support disaster recovery for business workflow across a distributed network of application and data servers. The IBM zSeries eServers, are enterprise class platform optimized for integration and designed to handle the transaction and data demands of on demand. The zSeries servers include the IBM zSeries 800 (z800), zSeries 900 (z900), and zSeries 990 (z990), S/390® Parallel Enterprise Server™ Generation 5 and 6, the Multiprise® 3000 and two Linux mainframe solutions, the IBM Integrated Platform for on demand on zSeries and the IBM zSeries Offering for Linux.

Features of zSeries that benefit e-business The zSeries servers can be configured in numerous ways to offer unparalleled flexibility to speed deployment of on demand business solutions. The zSeries servers are based on IBM z/Architecture®, which supports a new standard Architecture, that supports a new standard of performance, capacity, and integration by expanding on the balanced system approach of the S/390 architecture. Virtually unlimited 64-bit addressing capability, providing extra capacity for zSeries offers high performance Gigabit Ethernet features. The zSeries improved the S/390 I/O subsystem to complement the increased number of processors and larger main memory of the zSeries. High-speed interconnects for TCP/IP communication, Hypersockets, let TCP/IP traffic travel between images at memory speed, rather than network speed. Virtual Local Area Network (VLAN) support for all of the OSA-Express features when configured in QDIO mode for the Linux environment. The zSeries Fibre Channel Protocol (FCP) channel enhances Linux environments to allow access through a fibre-channel fabric to storage devices connected to industry-standard Small Computer System Interface (SCSI) controllers. This allows customization of a zSeries FICON, or FICON Express feature as an FCP channel.

Other Advantages of zSeries The zSeries servers deliver the highest level of application availability required in the global networked environment today. Another aspect of availability is non-disruptive growth, enabled in the zSeries by IBM e Capacity Upgrade on demand. The zSeries and S/390 servers can add server capacity and virtual servers non-disruptively and install FICON, ESCON, and OSA-Express ATM, Gigabit, Fast Ethernet, and Token-Ring cards without bringing the system down. The upgrade can even be initiated by customers over the Internet. The zSeries with z/VM® provides flexibility and management characteristics that make it possible to satisfy the business on demand marketplace by deploying new Linux servers in minutes rather than days or weeks. The z/VM V4 is ideally suited for deploying Linux workloads on the mainframe. The z/VM still benefits from the attractive per-engine pricing schemea of z/VM V4 and continues to support the Integrated Facility for Linux (IFL). IFLs are processor engines that enable customers to add processor capacity to existing zSeries systems. The z/VM V4 provides additional support and exploitation opportunities for the thousands of users who have built enterprise-wide automation and infrastructure enhancements on the VM platform, in support of their applications, database systems, and on demand business solutions.

Virtual Server Services IBM launched its first hosting service on Linux running on zSeries mainframes, July 2002. Predicted savings of between 15 and 30 percent for customers using its virtual server service versus owning the machines outright are calculated. These hosting offerings are referred to as Virtual Server Service, which basically is a promise to host isolated environments and only charge each customer for the cycles they use, with a set-up fee and monthly “on demand” charges. These virtual servers are deployed at an IBM Service Delivery Center and managed by IBM Global Services and offer additional services, connection to the Internet, caching, storage, and backup services, as well as firewalls and load balancing (through a partnership with Inkra Networks).

Capacity on Demand On/Off Capacity On Demand lets customers unlock extra capacity to handle seasonal peaks in a business cycle, or to otherwise accommodate unplanned spikes in demand. The On/Off Capacity On Demand feature now also encompasses the Integrated Facility for Linux (IFL) engines running on zSeries hardware. There is also a new fibre channel attachment to ease Linux to mainframe consolidation efforts. Customers consolidating from distributed Linux systems to Big Iron Linux, no longer need to copy all of their data over to the mainframe. The new attachment allows them the option of leaving the data on existing open storage devices. This allows a customer to leverage existing storage investments while consolidating onto the mainframe.

Conclusion The four characteristics of an on demand business are responsiveness, focus, resilience, and variability. Using open standards provides the advantage of facilitating as varied an infrastructure as possible, and providing the greatest opportunities for innovation. Standards in general allow for integration and virtualization of resources, which frees users from having to understand how components work. They also permit for the formation of computing grids that allow an enterprise to share computing resources, thus optimizing the installed base of information technology. The zSeries servers are ideal candidates to be in an on demand business platform. They can be configured in numerous ways to offer unparalleled flexibility to speed deployment of on demand solutions. They are also renown for there degree of availability and security, as well as performance, capacity, and integration opportunities. They offer 64-bit addressing capability, providing extra capacity Gigabit Ethernet features, increased number of processors and larger main memory and high-speed interconnects for TCP/IP communication. Finally, the benefits of using z/VM zSeries facilitates non-disruptive growth that allows a client to request new virtual machines to deal with increased workloads, without bringing down the system or affecting availability.

Conclusion Virtual Server Services are hosting offerings that is a promise to host isolated environments and only charge each customer for the cycles they use, with a set-up fee and monthly “on demand” charge. On/Off Capacity On Demand lets customers unlock extra capacity to handle seasonal peaks in a business cycle, or to otherwise accommodate unplanned spikes in demand. “Grid” computing differs from regular distributed computing due to its focus on large-scale resource sharing, innovative applications, and, high-performance orientation. This sharing between organizations creates unique authentication, authorization, resource access, resource discovery, and other technical challenges. The need is to achieve various qualities of service when running on top of different native platforms. Grid architecture is composed of five layers: application, collective, resource, connectivity, fabric each with associated protocols, and open standards that make it flexible.

Conclusion Open Grid Services Architecture (OGSA) is a combination of Grid and Web service technologies. OGSA uses the Web Services Description Language (WSDL) to achieve self-describing, discoverable services and interoperable protocols, with extensions to support multiple coordinated interfaces and change management. OGSA utilizes the advantages of the Globus Toolkit to define conventions and WSDL interfaces for a Grid service, a service instance supporting reliable and secure invocation, lifetime management, notification, policy management, credential management, and virtualization. Grid services are contained within an execution or hosting environment that defines the implementation programming model, programming language, development tools, and debugging tools. Hosting environment also describes how an implementation of a Grid service will meet its obligations with respect to Grid service semantics.