COMP3019 Coursework: Introduction to M-grid Steve Crouch School of Electronics and Computer Science.

Slides:



Advertisements
Similar presentations
Computer networks Fundamentals of Information Technology Session 6.
Advertisements

University of Southampton Electronics and Computer Science M-grid: Using Ubiquitous Web Technologies to create a Computational Grid Robert John Walters.
Setting up of condor scheduler on computing cluster Raman Sehgal NPD-BARC.
Introduction CSCI 444/544 Operating Systems Fall 2008.
M-grid Using Ubiquitous Web Technologies to create a Computational Grid R J Walters and S Crouch 21 January 2009.
Condor and GridShell How to Execute 1 Million Jobs on the Teragrid Jeffrey P. Gardner - PSC Edward Walker - TACC Miron Livney - U. Wisconsin Todd Tannenbaum.
Introduction to Operating Systems CS-2301 B-term Introduction to Operating Systems CS-2301, System Programming for Non-majors (Slides include materials.
Web Servers How do our requests for resources on the Internet get handled? Can they be located anywhere? Global?
Introduction  What is an Operating System  What Operating Systems Do  How is it filling our life 1-1 Lecture 1.
The Open Grid Service Architecture (OGSA) Standard for Grid Computing Prepared by: Haoliang Robin Yu.
Condor Overview Bill Hoagland. Condor Workload management system for compute-intensive jobs Harnesses collection of dedicated or non-dedicated hardware.
Distributed Computer Architecture Benjamin Jordan, Kevin Cone, Jason Bradley.
Distributed Computing Overviews. Agenda What is distributed computing Why distributed computing Common Architecture Best Practice Case study –Condor –Hadoop.
Lecture # 1 M.Sc / BS(CS)/ BS(I.T) DPT www. Shahid Rasul.com.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
For more notes and topics visit:
Alain Roy Computer Sciences Department University of Wisconsin-Madison An Introduction To Condor International.
Distributed Systems Early Examples. Projects NOW – a Network Of Workstations University of California, Berkely Terminated about 1997 after demonstrating.
Grid Toolkits Globus, Condor, BOINC, Xgrid Young Suk Moon.
Cloud Computing for the Enterprise November 18th, This work is licensed under a Creative Commons.
Introduction to Computers
SICSA student induction day, 2009Slide 1 Social Simulation Tutorial Session 6: Introduction to grids and cloud computing International Symposium on Grid.
Track 1: Cluster and Grid Computing NBCR Summer Institute Session 2.2: Cluster and Grid Computing: Case studies Condor introduction August 9, 2006 Nadya.
Connecting OurGrid & GridSAM A Short Overview. Content Goals OurGrid: architecture overview OurGrid: short overview GridSAM: short overview GridSAM: example.
Introduction To Computer System
Jaeki Song ISQS6337 JAVA Lecture 16 Other Issues in Java.
Introduction and Overview Questions answered in this lecture: What is an operating system? How have operating systems evolved? Why study operating systems?
M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.
CS 1308 Computer Literacy and the Internet. Introduction  Von Neumann computer  “Naked machine”  Hardware without any helpful user-oriented features.
Grid Computing - AAU 14/ Grid Computing Josva Kleist Danish Center for Grid Computing
GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.
Web: Towards Grid Interoperability Richard Boardman, Stephen Crouch, Hugo Mills, Steven Newhouse, Juri Papay and.
Grids and Portals for VLAB Marlon Pierce Community Grids Lab Indiana University.
CSE 548 Advanced Computer Network Security Document Search in MobiCloud using Hadoop Framework Sayan Cole Jaya Chakladar Group No: 1.
Grid Computing I CONDOR.
COMP3019 Coursework: Introduction to GridSAM Steve Crouch School of Electronics and Computer Science.
1 Overview of the Application Hosting Environment Stefan Zasada University College London.
Grid Technologies  Slide text. What is Grid?  The World Wide Web provides seamless access to information that is stored in many millions of different.
Operating System Concepts Chapter One: Introduction What is an operating system? Simple Batch Systems Multiprogramming Systems Time-Sharing Systems Personal-Computer.
Privilege separation in Condor Bruce Beckles University of Cambridge Computing Service.
Operating Systems David Goldschmidt, Ph.D. Computer Science The College of Saint Rose CIS 432.
1 MSCS 237 Overview of web technologies (A specific type of distributed systems)
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
SEE-GRID-SCI The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no.
World Wide Web “WWW”, "Web" or "W3". World Wide Web “WWW”, "Web" or "W3"
Module: Software Engineering of Web Applications Chapter 2: Technologies 1.
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
Condor Services for the Global Grid: Interoperability between OGSA and Condor Clovis Chapman 1, Paul Wilson 2, Todd Tannenbaum 3, Matthew Farrellee 3,
Hands-On Microsoft Windows Server 2008 Chapter 5 Configuring Windows Server 2008 Printing.
Selenium server By, Kartikeya Rastogi Mayur Sapre Mosheca. R
Chapter 7 Operating Systems Foundations of Computer Science  Cengage Learning 1.
PARALLEL AND DISTRIBUTED PROGRAMMING MODELS U. Jhashuva 1 Asst. Prof Dept. of CSE om.
September 2003, 7 th EDG Conference, Heidelberg – Roberta Faggian, CERN/IT CERN – European Organization for Nuclear Research The GRACE Project GRid enabled.
E-commerce Architecture Ayşe Başar Bener. Client Server Architecture E-commerce is based on client/ server architecture –Client processes requesting service.
Discovering Computers 2009 Chapter 1 Introduction to Computers.
Progress Apama Fundamentals
Accessing the VI-SEEM infrastructure
Introduction to Computers
Chapter 1: Introduction
The Open Grid Service Architecture (OGSA) Standard for Grid Computing
GWE Core Grid Wizard Enterprise (
Grid Computing.
University of Technology
Chapter 1: Introduction
Lecture # 1 M.Sc / BS(CS)/ BS(I.T) DPT.
湖南大学-信息科学与工程学院-计算机与科学系
Basic Grid Projects – Condor (Part I)
LO3 – Understand Business IT Systems
Credential Management in HTCondor
Presentation transcript:

COMP3019 Coursework: Introduction to M-grid Steve Crouch School of Electronics and Computer Science

Objectives  To equip students to drive a lightweight grid implementation to solve a problem that can benefit from using grid technology.  To develop an understanding of the basic mechanisms used to solve such problems.  To develop a general architectural and operational understanding of typical production-level grid software.  To develop the programming skills required to drive typical services on a production- level grid.

Overview  Part 1: m-grid – m-grid: lightweight software illustrating grid concepts in use – Develop a program with m-grid’s Java API to solve a simple problem, submit it to m-grid with input data, collect results  Part 2: Google MapReduce & GridSAM – MapReduce: framework for distributed processing of large datasets using many computers – GridSAM: job submission web service interface to a computational resource (e.g. compute cluster, single machine) – Extend code stubs to submit jobs to GridSAM and monitor them to completion – Extend pseudocode that implements a basic MapReduce framework

Where to get stuff/help?  Can obtain coursework materials from website – Ready for Wednesday  Software documentation  Coursework help lecture 19 th March  Myself:  Building 32: Level 4 lab 4067 Bay 23

Background

The Problem  Basically, want to run compute-intensive task  Don’t have enough resources to run job locally – At least, to return results within sensible timeframe  Would like to use another, more capable resource

Distributed Computing in Olden Times Small number of ‘fast’ computers – Very expensive – Centralised – Used nearly all the time – Time allocations for users – Not updated often brewbooks Cray X-MP (Cray -1 successor) Univac 1710 Michael L. Umbricht and Carl R. Friend Punched cards Wait time huge MailNet, SneakerNet, TyperNet, etc… Mainframes Cray $8.8 million, 160 megaflops, 8MB memory

The Present…  Now… large number of slow computers: – Cheap – Distributed  Computation  Ownership – Not used all the time – Exclusive access to users – Updated often – e.g. desktop computers, PDAs, mobile phones  Low utilisation of computing power  e.g.: institutional/university resources…

It’s About Scaling Up… Compute and data – you need more, you go somewhere else to get it Local Institutional National International Then… the march towards localisation of computation, the Personal Computer Computational Science develops in laboratories Is this changing again? Images: nasaimages, Extra Ketchup, Google Maps, Dave Page

The Grid - a Reminder  The grid – many definitions! “Grid computing offers a model for solving massive computational problems by making use of the unused CPU cycles of large numbers of disparate, often desktop, computers treated as a virtual cluster embedded in a distributed telecommunications infrastructure” – Wikipedia “A service for sharing computer power and data storage capacity over the Internet.“ – CERN (European Organisation for Nuclear Research)  Two components of grid computing: – Computational/data resource – e.g. computational cluster, supercomputer, desktop machine – Infrastructure for externalising that resource to others

Some Examples…  Grid (i.e. internet-accessible) examples: – -  Process data from Arecibo Radio Telescope, Puerto Rico  2 million volunteers installed software – Univa.org-  Projects such as Cancer Research, Smallpox  2.5 million volunteer systems  Sells processing time to organisations  Computational resource (i.e. intranet- accessible): – Cluster managers, supercomputer, single machine

The Idea - as a Provider…  Goal: I want others to access my resources & applications  I want to provide secure controlled access to: – My applications:  Specify who can access which applications – My computational or data resources:  I can limit external usage of my resources  Provides an interface that allows remote users to access my resources  Enable collaboration with other partners

The Idea - as a User (or Client)  Goal: I want to use other resources & applications  Through a network of service providers I can…: – Gain access to applications that I do not have installed locally – Use remote machines [larger resource] with more CPU, memory or storage  Process larger problem sizes – Transparently switch between different service providers  No exposure to underlying OS, queuing policy, disk layout etc.

Grid Architecture  The best way of designing grids… – Loosely coupled services – Message based exchange  The best way of running grids… – Interoperability between versions & grids – Standards for infrastructure & services  The best way of building grids… – Leverage existing infrastructure & standards – Use Web Services…

Cluster Computing & the Grid Grid is predominantly built on Cluster Computing solutions University B University A University C Grid Cluster Computing

The General Idea…  Abstract ‘virtualisation’ of local network resources – Infrastructure manages many machines – Visualisation as a single resource – Submitted jobs get put on queue(s) CoordinatorExecutor Client … …

Condor – Background  Begun in 1988, based on Remote-Unix (RU) project  Predominantly makes use of idle cycles on machines

Condor Components  Four main machine ‘roles’ (daemons): – Submit Client (condor_schedd): used to submit resource requests, monitor, modify and delete jobs. – Central Manager, Server  condor_collector: collects information about pool resources.  condor_negotiator: negotiates (match-makes) between resources and resource requests. – Job Executor (condor_startd): executes jobs, advertises resources. Enforces local policy. – (Checkpoint Server (condor_ckpt_server): services requests to store and retrieve checkpoint files.)

Other Condor daemons…  condor_master: controls daemons running on a machine. Keeps daemons running depending on machine role.  condor_shadow: monitors running job on behalf of submit client, from submit client’s machine. i.e. one daemon for every client job.  condor_starter: creates environment, then runs and monitors job on executor. i.e. one daemon for every executor job.

Shared Disk Condor Architecture 1.Client submits job (executable + input data) to local queue 2.Client schedd advertises job request to server collector 3.Server negotiator gets next priority request from collector 4.Negotiator negotiates w/ client schedd to match resource/job 5.Client removes job from queue and sends it to executor 6.Job runs on executor 7.Job output results returned to client Client Server Executor Submit client (condor_schedd, condor_shadow…) Negotiator (condor_negotiator) Collector (condor_collector) Executor (condor_startd, condor_starter…) Queue … … 1 7

M-grid An overview

Computational Grids - in General  Users supply tasks to be performed via client  Execution nodes contribute processing power  Coordinator node sends tasks to execution nodes, ensuring results returned  Existing grid tech. sophisticated -> significant complexity – To what extent can this be reduced? CoordinatorExecutor Client … …

Java Applets?  How about Java applets as a program unit? – Browsers could act as execution nodes  Security concerns? – Web browsers execute foreign code – Java applets executed within a ‘sandbox’ virtual machine – Stringent security restrictions imposed – In-built security configuration in browsers – Applet can only contact originating server  Risk significantly reduced

M-grid: A Lightweight Grid I  M-grid: – Execution node = Java-applet enabled browser – Client = browser – Coordinator = web server – Tasks distributed as Applets in web pages  Execution node browser opens web page on server  Runs task applet  Uploads results to server CoordinatorExecutor Client ……

M-grid: Overview  Implemented on: – Microsoft’s IIS (Internet Information Server) using ASP – Apache Tomcat – we’ll use this one!  Client – Develops applet class as extension to MGridApplet class – Can run applet locally in appletviewer for testing – Compiles and packages applet with input parameters file into a jar file – Submits jar to web server via JobSubmit web page – Eventually collects results via ViewJobs web page  Execution node – Requests a job via JobRequest page – Applet submits results from job using SubmitResults page  Security provided by session authentication

Architecture