The Grid ”Enter the GRID” af Kristian Mandrup

Indeks Intro Overview Architecture Solutions Future Conclusions & discussion

What is it ? The Next-Gen Internet? A 21st century time machine?

Intro Future of collaborative problem-solving Internet's next evolutionary step The Grid is a new class of infrastructure Link computers in new ways Open up storage and transaction power as Web opened up content

Intro (2) Era of distributed, networked computing is just beginning The WWW a taste, the Grid a vision Answer to the enterprise computing crisis (ECC)

Vision Applies interconnected model used by power utilities to access services, software and hardware resources as part of virtual supercomp. Executes jobs on best suited, least loaded systems in a seamless, transparent and secure way On-demand access to computational power, data bases and services Manage resource sharing and co-ordinated problem solving across dynamic, multi-institutional virtual organisations both in eScience and eBusiness Provides scalable, secure, high-performance mechanisms for discovering and negotiating access to remote resources Geographically distributed groups can work together in new ways

Background (evolution) Breakthrough technologies Begun in the research environment Moved to open standards Applied to business applications What we are seeing with Grid standards

Background (history) Desire to connect supercomputers into "metacomputers" that could be remotely controlled Vision of the Grid started in 1960s Envisioned a computer facility operating "like a power company or water company" Word "grid" borrowed from the electricity grid Any compatible device could be plugged in anywhere on the Grid and be guaranteed a certain level of resources, regardless of where those resources might come from

Evolution 1G Grids Involved local "meta-computers" with basic services such as distributed file systems and site-wide single sign on. 1G Grids were totally custom made 2G Grids Underlying software services and communications protocols Grids offered basic building blocks, but deployment involved significant customization Interoperability among 2G Grid systems very difficult 3G Grids Solves deployment and interoperability issues by providing standard interfaces Today it feasible to realize the Grid vision Global Grid Forum (GGF) created in November, 2000

Demand Science & Industry High-energy physics, needs extra resources to manage and analyze huge amounts of data Science and industry participants require level of reliability not offered by current peer-to-peer initiatives Strong need to efficiently manage availability of distributed infrastructures, applications and services Computational resources are failing to keep up with what scientists demand of them

Demand (technical) Doubling periods (months) Network bandwidth 9 Storage capacity 12 Computing power 18 Computer power is falling behind storage !

Demand (example) Scientists create high-resolution simulations need petabyte archives CERN's Large Hadron Collider (LHC) will produce multiple petabytes (10 15 byte) of data per year Scientists demand 10+ Gb/s to work remotely on petabyte data sets Law of diminishing returns ???

Demand (solutions) If communication is unlimited and free Not restricted to using local resources to solve problems Use collective computing power of research collaboration or buy from provider Look at large datasets using special collaboration and visualization tools Use remote resources to do things not possible using local resources

Benefits Aggregates compute power and delivers it as a network service Grid Engine presents users to a seamless, integrated computing capability Facilitate the deployment of compute farms, the basic building blocks of grid computing Making large amounts of compute power available for applications and users

Benefits ”sales talk”  Raise productivity  Maintain availability  Minimize downtime  Shorter time to market  Reduces costs by better utilisation of resources  Quicker and better results  Increased quality and innovation  Do things not possible before  Increased ROI (Return On Investment)

Potential problems Social and political dimensions (like WWW) Sharing between strangers where no history of trust

Uses  Development of semiconductors  Bioinformatics  Mechanical design  Software development  Oil/gas exploration  Financial analysis  Academic and research pursuits

Architecture (Infrastructure) Open Grid Services Architecture (OGSA) Integration of Grid and Web services technologies Open Grid Services Infrastructure (OGSI) Grid Resource Access and Management (GRAM) protocol and service Remote resource allocation and process creation Monitoring Management services

Architecture (OGSA) Open Grid Services Architecture  Establish standard interfaces and behaviours for distributed system management  Management of service instances (persistent or transient)  Defines fundamental WDSL interfaces: to establish a Grid service in the open source Global Toolkit 3.0 (GT3)  Grid service instance: maintains a set of service data elements by encapsulating XML fragments in standard containers  FindServiceData operation: queries this information and allows notification of service existence and modifications in service  Includes GT3 (Global Toolkit 3) Core and Base Services

Architecture 1) physical devices or resources 2) Core communication and authentication protocols cryptographically secure mechanisms - verifying identity of users and resources 3) Protocols, services, and APIs  Implement interactions across collections of resources  Directory and brokering services for resource discovery and allocation  Monitoring and diagnostic services  Data replication services  Membership and policy services 4) User applications

Security Unlike the Web, the Grid is being designed from the ground up as a secure system Accept only messages coming from special hosts and reserved ports Integration with Kerberos5 and DCE exists Authentication, authorization, and policy  Client and a server need to mutually authenticate each other.  No distinction between client and server. Server one moment, client another moment.  Special requirements for managing transaction

Security (method) Single sign-on: Via creation of a proxy credential Mapping to local security mechanisms: Grid security infrastructure maps to local solutions at each site Delegation: Sub-computations created at sites A and B. Both communicate with each other and access files at site C Community authorization and policy: infeasible for each resource to keep track of community membership and privileges. Group membership identified with cryptographic credential issued by trusted third party

Security (how it works)  User calls on computational resources of sites A and B  Communicate with each other, read files located at site C.  Each step requires authorization and authentication  Mediating requests requires the Grid Security Infrastructure (GSI)  Provides:  Single sign-on  Run-anywhere authentication service  Support for delegation of credentials to sub-computations  Local control over authorization  Mapping from global to local user identities

Implementation requirements Implementing architecture requires uniform mechanisms  Creating and managing services on remote computers  Supporting single sign-on to distributed resources  Transferring large datasets at high speed  Forming large distributed virtual communities  Maintaining information about existence, state, and usage policies of community resources

Solutions Sun Microsystems acquired Gridware, a private developer of Distributed Resource Management (DRM) software, in July 2000 Becomes Grid Engine project Grid Engine project goals: New open standards for DRM Standard API for application integration Grid Engine Portal (GEP) Provides a Java based capability for enabling highly secure internet access to applications that run on an existing Grid Engine grid Loosely coupled to Grid Engine, SunONE Portal Server Globus Toolkit (1996) Standards-based protocols for distributed system management for open source implementation

Using the Grid Steps to take  Discover resources exist.  Negotiate access to resources  Configure hardware/software to use resources  Avoid compromising security of self or remote resources

How it works Obtaining: authentication credentials Querying: Information system and replica catalog to determine availability of computers, storage systems, and networks, and location of required input files (collective services) Submitting: requests to appropriate computers, storage systems, and networks to initiate computations, move data, and so forth (resource protocols) Monitoring: the progress of the various computations and data transfers, notifying the user when all are completed, and detecting and responding to failure conditions (resource protocols)

Status Grid Engine software has been ported to many operating systems, including Solaris, Linux Current v.5.4 can be downloaded

The Future The Grid may give birth to a global file- swapping network or a members-only citadel for moneyed institutions The future of the Grid is unknown !

Conclusions & discussions Is this the distributed systems utopia ? Is anything missing ? What is the next step after ”the Grid” – ”the Matrix” ??? What is ”the Matrix” ? CM-systemers rolle ?