Grid Computing Environments Grid: a system supporting the coordinated resource sharing and problem-solving in dynamic, multi-institutional virtual organizations.

Slides:



Advertisements
Similar presentations
GT 4 Security Goals & Plans Sam Meder
Advertisements

The Anatomy of the Grid: An Integrated View of Grid Architecture Carl Kesselman USC/Information Sciences Institute Ian Foster, Steve Tuecke Argonne National.
High Performance Computing Course Notes Grid Computing.
1.1 © 2004 Pearson Education, Inc. Exam Managing and Maintaining a Microsoft® Windows® Server 2003 Environment Lesson 1: Introducing Windows Server.
Grid computing Globus GridFTP & Replica Management Robert Nickel BTU - Mathematik 01.Februar 2002.
GridFTP Introduction – Page 1Grid Forum 5 GridFTP Steve Tuecke Argonne National Laboratory.
A Computation Management Agent for Multi-Institutional Grids
Seminar Grid Computing ‘05 Hui Li Sep 19, Overview Brief Introduction Presentations Projects Remarks.
Resource Management of Grid Computing
USING THE GLOBUS TOOLKIT This summary by: Asad Samar / CALTECH/CMS Ben Segal / CERN-IT FULL INFO AT:
Introduction to Grid Computing The Globus Project™ Argonne National Laboratory USC Information Sciences Institute Copyright (c)
Grid Computing 7700 Fall 2005 Lecture 5: Grid Architecture and Globus Gabrielle Allen
1-2.1 Grid computing infrastructure software Brief introduction to Globus © 2010 B. Wilkinson/Clayton Ferner. Spring 2010 Grid computing course. Modification.
Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.
Milos Kobliha Alejandro Cimadevilla Luis de Alba Parallel Computing Seminar GROUP 12.
Introduction to Grid Computing and the Globus Toolkit™
4b.1 Grid Computing Software Components of Globus 4.0 ITCS 4010 Grid Computing, 2005, UNC-Charlotte, B. Wilkinson, slides 4b.
1 GRID D. Royo, O. Ardaiz, L. Díaz de Cerio, R. Meseguer, A. Gallardo, K. Sanjeevan Computer Architecture Department Universitat Politècnica de Catalunya.
Grids and Globus at BNL Presented by John Scott Leita.
Globus Computing Infrustructure Software Globus Toolkit 11-2.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Resource Management Reading: “A Resource Management Architecture for Metacomputing Systems”
Grid Toolkits Globus, Condor, BOINC, Xgrid Young Suk Moon.
GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.
1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong
Grid Resource Allocation and Management (GRAM) Execution management Execution management –Deployment, scheduling and monitoring Community Scheduler Framework.
The Anatomy of the Grid: An Integrated View of Grid Architecture Ian Foster, Steve Tuecke Argonne National Laboratory The University of Chicago Carl Kesselman.
The Grid System Design Liu Xiangrui Beijing Institute of Technology.
The Anatomy of the Grid Mahdi Hamzeh Fall 2005 Class Presentation for the Parallel Processing Course. All figures and data are copyrights of their respective.
Virtual Data Grid Architecture Ewa Deelman, Ian Foster, Carl Kesselman, Miron Livny.
1 4/23/2007 Introduction to Grid computing Sunil Avutu Graduate Student Dept.of Computer Science.
Globus Replica Management Bill Allcock, ANL PPDG Meeting at SLAC 20 Sep 2000.
Introduction to Grid Computing Ed Seidel Max Planck Institute for Gravitational Physics
June 24-25, 2008 Regional Grid Training, University of Belgrade, Serbia Introduction to gLite gLite Basic Services Antun Balaž SCL, Institute of Physics.
Ames Research CenterDivision 1 Information Power Grid (IPG) Overview Anthony Lisotta Computer Sciences Corporation NASA Ames May 2,
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
GRID ARCHITECTURE Chintan O.Patel. CS 551 Fall 2002 Workshop 1 Software Architectures 2 What is Grid ? "...a flexible, secure, coordinated resource- sharing.
Data Management and Transfer in High-Performance Computational Grid Environments B. Allcock, J. Bester, J. Bresnahan, A. L. Chervenak, I. Foster, C. Kesselman,
GRIDS Center Middleware Overview Sandra Redman Information Technology and Systems Center and Information Technology Research Center National Space Science.
CEOS Working Group on Information Systems and Services - 1 Data Services Task Team Discussions on GRID and GRIDftp Stuart Doescher, USGS WGISS-15 May 2003.
Globus – Part II Sathish Vadhiyar. Globus Information Service.
Introduction to Grids By: Fetahi Z. Wuhib [CSD2004-Team19]
1 Observations on Architecture, Protocols, Services, APIs, SDKs, and the Role of the Grid Forum Ian Foster Carl Kesselman Steven Tuecke.
Globus and PlanetLab Resource Management Solutions Compared M. Ripeanu, M. Bowman, J. Chase, I. Foster, M. Milenkovic Presented by Dionysis Logothetis.
Globus Grid Tutorial Part 2: Running Programs Across Multiple Resources.
Securing the Grid & other Middleware Challenges Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer.
GraDS MacroGrid Carl Kesselman USC/Information Sciences Institute.
GRID ANATOMY Advanced Computing Concepts – Dr. Emmanuel Pilli.
1 I.Foster LCG Grid Technology: Introduction & Overview Ian Foster Argonne National Laboratory University of Chicago.
1 Active Directory Service in Windows 2000 Li Yang SID: November 2000.
Globus: A Report. Introduction What is Globus? Need for Globus. Goal of Globus Approach used by Globus: –Develop High level tools and basic technologies.
Globus Data Storage Interface (DSI) - Enabling Easy Access to Grid Datasets Raj Kettimuthu, ANL and U. Chicago DIALOGUE Workshop August 2, 2005.
Rights Management for Shared Collections Storage Resource Broker Reagan W. Moore
Protocols and Services for Distributed Data- Intensive Science Bill Allcock, ANL ACAT Conference 19 Oct 2000 Fermi National Accelerator Laboratory Contributors:
Middleware and the Grid Steven Tuecke Mathematics and Computer Science Division Argonne National Laboratory.
The Globus Toolkit The Globus project was started by Ian Foster and Carl Kesselman from Argonne National Labs and USC respectively. The Globus toolkit.
Active Directory Domain Services (AD DS). Identity and Access (IDA) – An IDA infrastructure should: Store information about users, groups, computers and.
E-commerce Architecture Ayşe Başar Bener. Client Server Architecture E-commerce is based on client/ server architecture –Client processes requesting service.
INTRODUCTION TO GRID & CLOUD COMPUTING U. Jhashuva 1 Asst. Professor Dept. of CSE.
Peter Kacsuk – Sipos Gergely MTA SZTAKI
Globus —— Toolkits for Grid Computing
University of Technology
Grid Computing B.Ramamurthy 9/22/2018 B.Ramamurthy.
The Globus Toolkit™: Information Services
Distributed Systems Bina Ramamurthy 11/30/2018 B.Ramamurthy.
Distributed Systems Bina Ramamurthy 12/2/2018 B.Ramamurthy.
Introduction to Grid Technology
The Anatomy and The Physiology of the Grid
The Anatomy and The Physiology of the Grid
Distributed Systems Bina Ramamurthy 4/22/2019 B.Ramamurthy.
Presentation transcript:

Grid Computing Environments Grid: a system supporting the coordinated resource sharing and problem-solving in dynamic, multi-institutional virtual organizations. Note: some of this material was taken from the Globus tutorial at

A Scenario A fundamental change in the problems needed to be solved: multidisciplinary nature  composition of expertise larger problem scale  composition of computing resources Mary is a university researcher in North America who is collaborating with two colleagues - Hans at a corporate R&D facility in Europe and Ling at a research institute in Asia. Mary has a new research model that she would like to validate using data collected by Ling and compare the results to that of the proprietary model developed by Hans for his corporation. Each of the three collaborators have previously constructed software components that provide access to the resources they are contributing to the collaboration. Mary and Hans are using a shared workspace system to interact synchronously in performing the computations, viewing the results, and saving the results for later study. Ling is not available at the time of the interaction and needs to provide appropriate access so that her data is available to Mary and Hans. The results, stored by Mary, should also be available later for study by Ling.

Grid Computing Environments Virtual Organization: a set of individuals and/or institutions collaborating to achieve a common goal within a set of rules defining the controlled sharing of computational resources. Characteristics of virtual organizations: flexible relationships (client-server, P2P, bockered,…) complex sharing rules (access control, delegation,…) varied resources (programs, storage, devices, …) diverse usage modes (single vs. multi-user, performance vs. cost sensitive, synchronous vs. asynchronous,…)

On-demand creation of powerful virtual organizations Grid vs.Web Web: Uniform naming/access to documents Grid: Uniform, high- performance access to computational resources Sensor nets Data archives Computers Software catalogs Colleagues

Grid vs. Web CharacteristicsGridWeb Seamless namingYes Uniform security/authentication YesNo Information servicesYesYes/no Co-schedulingYesNo Accounting/authorizationYesNo User servicesYesNo Event servicesYesNo Global shellyesno

Layered Grid Architecture Application Fabric “Controlling things locally”: Access to, & control of, resources Connectivity “Talking to things”: communication (Internet protocols) & security Resource “Sharing single resources”: negotiating access, controlling use Collective “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services Internet Transport Application Link Internet Protocol Architecture

“Hourglass” principle Application Fabric Connectivity Resource Collective Local OS Core Grid services

Protocols, Services,and Intefaces Languages/Frameworks Fabric Layer Applications Local Access APIs and Protocols Collective Service APIs and SDKs Collective Services Collective Service Protocols Resource APIs and SDKs Resource Services Resource Service Protocols Connectivity APIs Connectivity Protocols

Protocols, Services, Interfaces: Protocol-mediated access to resources –Mask local heterogeneities –Extensible to allow for advanced features –Negotiate multi-domain security, policy –“Grid-enabled” resources speak protocols –Multiple implementations are possible Broad deployment of protocols facilitates creation of Services that provide integrated view of distributed resources Intefaces (APIs/SKDs) use protocols and services to enable specific classes of applications

Globus NSF’s National Technology Grid NASA’s Information Power Grid Internet protocols Globus Security Infrastructure (GSI) Grid Resource Information Service (GRIS) Grid Resource Access and Management (GRAM) GridFTP Grid Information Index Servers (GIIS) Replica management Certificate repository (MyProxy) Co-allocation library (DUROC) Fabric Connectivity Resource Collective

Fabric Layer A diverse mix of resources that may be shared –Individual computers, Condor pools, file systems, archives, metadata catalogs, networks, sensors, etc., etc. Few constraints on low-level technology: connectivity and resource level protocols form the “neck in the hourglass” Defined by interfaces not physical characteristics

Connectivity Layer Communication –Internet protocols: IP, DNS, routing, etc. Security: Grid Security Infrastructure (GSI) –Uniform authentication & authorization mechanisms in multi-institutional setting –Single sign-on, delegation, identity mapping –Public key technology, SSL, X.509, GSS-API –Supporting infrastructure: Certificate Authorities, key management, etc.

Why Grid Security is Hard Resources being used may be extremely valuable & the problems being solved extremely sensitive Resources are often located in distinct administrative domains –Each resource may have own policies & procedures Set of resources used by a single computation may be large, dynamic, and unpredictable –Not just client/server It must be broadly available & applicable –Standard, well-tested, well-understood protocols –Integration with wide variety of tools

1) Easy to use 2) Single sign-on 3) Run applications ftp,ssh,MPI,Condor,Web,… 4) User based trust model 5) Proxies/agents (delegation) User View 1) Specify local access control 2) Auditing, accounting, etc. 3) Integration w/ local system Kerberos, AFS, license mgr. 4) Protection from compromised resources Resource Owner View API/SDK with authentication, flexible message protection, flexible communication, delegation,... Direct calls to various security functions (e.g. GSS-API) Or security integrated into higher-level SDKs: E.g. GlobusIO, Condor-G, MPICH-G2, HDF5, etc. Developer View Grid Security Requirements

Secure Remote Startup key cert gatekeeperclient 1. Exchange certificates, authenticate, delegate 2. Check gridmap file 3. Lookup service 4. Run service program (e.g. jobmanager) jobmanager key cert map 4. services 3.

Resource Layer Grid Resource Allocation Mgmt (GRAM) –Remote allocation, reservation, monitoring, control of compute resources GridFTP protocol (FTP extensions) –High-performance data access & transport Grid Resource Information Service (GRIS) –Access to structure & state information Network reservation, monitoring, control All integrated with GSI: authentication, authorization, policy, delegation

Metacomputing Directory Services Resources run a standard information service (GRIS) which speaks LDAP and provides information about the resource (no searching). GIIS provides a “caching” service much like a web search engine. Resources register with GIIS and GIIS pulls information from them when requested by a client and the cache as expired. GIIS provides the collective-level indexing/searching function. GIIS Cache contains info from A and B Resource A GRIS GIIS requests information from GRIS services as needed. Client 1 Client 2 Client 3 Resource B GRIS Clients 1 and 2 request info directly from resources. Client 3 uses GIIS for searching collective information.

Resource Management The Globus Resource Allocation Manager (GRAM) protocol and client API allows programs to be started on remote resources, despite local heterogeneity Resource Specification Language (RSL) is used to communicate requirements A layered architecture allows application-specific resource brokers and co-allocators to be defined in terms of GRAM services –Integrated with Condor, PBS, MPICH-G2, …

GRAM Components Grid Security Infrastructure Job Manager GRAM client API calls to request resource allocation and process creation. MDS client API calls to locate resources Query current status of resource Create RSL Library Parse Request Allocate & create processes Process Monitor & control Site boundary ClientMDS: Grid Index Info Server Gatekeeper MDS: Grid Resource Info Server Local Resource Manager MDS client API calls to get resource info GRAM client API state change callbacks

GridFTP Suite of communication libraries and related tools that support –GSI, Kerberos security –Third-party transfers –Parameter set/negotiate –Partial file access –Reliability/restart –Large file support –Data channel reuse All based on a standard, widely deployed protocol –Integrated instrumentation –Loggin/audit trail –Parallel transfers –Striping (cf DPSS) –Policy-based access control –Server-side computation –Proxies (firewall, load bal)

GridFTP Why FTP? –Ubiquity enables interoperation with many commodity tools –Already supports many desired features, easily extended to support others –Well understood and supported We use the term GridFTP to refer to –Transfer protocol which meets requirements –Family of tools which implement the protocol Note GridFTP > FTP

GridFTP: Basic Approach FTP protocol is defined by several IETF RFCs Start with most commonly used subset –Standard FTP: get/put etc., 3 rd -party transfer Implement standard but often unused features –GSS binding, extended directory listing, simple restart Extend in various ways, while preserving interoperability with existing servers –Striped/parallel data channels, partial file, automatic & manual TCP buffer setting, progress monitoring, extended restart

Collective Layer Index servers aka metadirectory services –Custom views on dynamic resource collections assembled by a community Resource brokers (e.g., Condor Matchmaker) –Resource discovery and allocation Replica catalogs Co-reservation and co-allocation services

Replica Management Maintain a mapping between logical names for files and collections and one or more physical locations Important for many applications –Example: CERN HLT data Multiple petabytes of data per year Copy of everything at CERN (Tier 0) Subsets at national centers (Tier 1) Smaller regional centers (Tier 2) Individual researchers will have copies

DUROC Simultaneous allocation of a resource set –Handled via optimistic co-allocation based on free nodes or queue prediction –In the future, advance reservations will also be supported globusrun will co-allocate specific multi- requests –Uses a Globus component called the Dynamically Updated Request Online Co-allocator (DUROC)