1 Globus Grid Middleware: Basics, Components, and Services Source: The Globus Project Argonne National Laboratory & University of Southern California Compiled/Compacted for : R. Buyya and S. Venugopal

2 Agenda The evolution of Globus Security Services GSI (Grid Security Infrastructure) Execution Management Services GRAM(Grid Resource Allocation and Management) Information Services MDS (Monitoring and Discovery System) Data Services GridFTP RFT (Reliable File Transfer) Summary/Conclusion

3 The Globus Project Basic research in grid-related technologies Resource & data management, security, QoS, policy, communication, adaptation, etc. Focused on creating a core Grid middleware Core services for grid-enabled tools & apps Developed Globus Toolkit (1.0 in 1998, 2.0 in 2001, 3.0 in 2004 based on web services/WSRF, 4.0 in 2008, and 5.0 in 2011) Construction of production grids & testbeds Multiple deployments to distributed organizations for production & prototyping Impacts The Southern California Earthquake Center uses Globus software to visualize earthquake simulation data.Southern California Earthquake Center Scientists in the Earth System Grid (ESG) are producing, archiving, and providing access to climate data that advances our understanding of global climate change.Earth System Grid (ESG) Find more on

Grid and Web Services: Convergence Grid Web The definition of WSRF(Web Services Resource Framework) means that Grid and Web communities can move forward on a common base WSRF Started far apart in apps & tech OGSI GT2 GT1 HTTP WSDL, WS-* WSDL 2 Have been converging Ack: Prof. Ian Foster, Globus Project

Open Grid Services Architecture Web Services Messaging, Security, Etc. Open Grid Services Infrastructure Domain-Specific Services Core Services Program Execution Data Services Standardization WS-Resource Framework Ack: Prof. Ian Foster, Globus Project

6 Globus Toolkit 4

7 Globus Toolkit Components Grid Resources and Local Services Grid Resource Management (GRAM, GASS) GSI Security Layer Third Party User-Level Middleware Grid Information Services (MDS) Grid Data Management (GridFTP, Replica Catalog) Applications Globus Grid Resources and Local Services Grid Resource Management (GRAM) GSI Security Layer Third Party User-Level Middleware Grid Information Services (MDS) Grid Data Management (GridFTP, Replica Catalog) Applications Globus

8 Security Services Review of Security Services

9 Grid Security Infrastructure (GSI) Based on Public key cryptography X.509 certificate Certificate/Proxy - Single Sign-on Gives information about the proxy $ grid-proxy-info Create proxy at /tmp/x509up_u Destroy proxy certificate $ grid-proxy-destroy

10 PKI: Creating Private & Public Keys To request a certificate a user starts by generating a key pair. The private key is stored encrypted with a pass phrase the user gives The public key is put into a certificate request Certificate Request Public Key Encrypted On local disk USER

11 PKI: Certificate Issuance The user then takes the certificate to the CA The CA usually includes a Registration Authority (RA) which verifies the request: The name is unique with respect to the CA It is the real name of the user Etc. Certificate Request Public Key ID

12 PKI: Certificate Issuance The CA then signs the certificate request and issues a certificate for the user. Certificate Request Public Key Name Issuer Public Key Signature Sign Rajkumar Buyya 111, Barry Street Carlton BD Male 165cms, 65Kg B&W Eyes State of Victoria Seal

13 Mutual Authentication (between A and B) A and B are two parties: Both need to trust each others’ CA. A  B (A establishes connection to B and gives his certificate (name,pub. Key) to B). B makes sure that it can trust CA of A. B generates random message  A and asks it encrypt it. A encrypts it and send to B B decrypts using A’s public key. If the msg. is same as what B has sent, then A is who it is claiming to be.

14 Example: Secure Remote Startup in Globus key cert gatekeeperclient 1. Exchange certificates, authenticate, delegate 2. Check gridmap file 3. Lookup service 4. Run service program (e.g. jobmanager) jobmanager key cert map 4. services 3.

15 Execution Management Services Grid Resource Allocation and Management (GRAM)

16 What is GRAM? Grid resource access & job execution services: Handles placement, provisioning and lifetime management of jobs GRAM is a unifying remote interface for remote job submission and resource management GRAM is for stateful job control Reliable operation Asynchronous monitoring and control

17 Grid Job Management Goals Provide a service to securely: Create an environment for a job Stage files to/from environment Cause execution of jobs Monitor execution of jobs Signal important state changes to client Enable client access to output files

18 GRAM Command Line Tools Pre-WS GRAM (GT2) Command Line Tools Interactive job submission globus-job-run host:port/jobmanager-pbs command Batch job submission globus-job-submit command Checking job status when done output globus-job-status jobID globus-job-output –out|err jobID Kill a job globus-job-cancel jobID

19 Grid Information Service MDS (Monitoring and Discovery Service)

20 The Need for Information System information is critical to operation of the grid and construction of applications How does an application determine what resources are available? What is the “state” of the computational grid? How can we optimize an application based on configuration of the underlying system? We need a general information infrastructure to answer these questions

21 Using Information for Resource Brokering “10 GFlops, EOS data, 20 Mb/sec -- for 20 mins” Metacomputing Directory Service GRAM Resource Broker Info service: location + selection Globus Resource Allocation Managers GRAM Fork LSF EASYLL Condor etc. “What computers?” “What speed?” “When available?” “50 processors + storage from 10:20 to 10:40 pm” “20 Mb/sec”

22 Examples of Useful Information Characteristics of a compute resource IP address, software available, system administrator, networks connected to, OS version, load Characteristics of a network Bandwidth and latency, protocols, logical topology Characteristics of the Globus infrastructure Hosts, resource managers(PBS, Condor etc)

23 Data Services Overview of Data Services

24 Data Services Handle movement, access, replication and updates of data Example services in Globus Toolkit GridFTP Reliable File Transfer (RFT)

25 Data Services - GridFTP Basic tool provided with Globus Toolkit globus-url-copy [-tcp-bs TCPBufSizeBytes] [-p parallelism] sourceURL destURL URL format = protocol://[host]:[port]/path Supported protocols gsiftp://, file://(on a local machine only), ftp://, It is legal to specify a user name and password in the the URL as follows n.com/file.dat

26 Data Services - GridFTP local -> local $ globus-url-copy gsiftp://localhost/etc/hosts file:///tmp/hosts_copy remote -> local $ globus-url-copy gsiftp://ng2.vpac.org/etc/hosts file:///tmp/hosts_copy local -> remote $ globus-url-copy file:///tmp/hosts_copy gsiftp://ng2.vpac.org/tmp

27 Data Services - Reliable File Transfer (RFT) Manages a set of third-party GridFTP transfers Uses a database to checkpoint transfer state Recovers from Source/destination server failures Network failures Container failures Transfers retried with exponential backoff Resumes transfers where they left off

28 Summary Globus as a core Grid middleware solution Four major services Security Service GSI Execution Management Service GRAM( Grid Resource Allocation and Management ) Information Service MDS (Monitoring and Discovery Service) Data Service GridFTP Reliable File Transfer Very prominent (during ) – used in several collaborative scientific computing and national grid projects

29 Some Issues with Globus Very complex system, need a strong administration skills to install/maintain Deep learning curve for developers who need to work with APIs More on Globus, visit: