Parallel Programming Models and Paradigms

Slides:



Advertisements
Similar presentations
A Workflow Engine with Multi-Level Parallelism Supports Qifeng Huang and Yan Huang School of Computer Science Cardiff University
Advertisements

MINJAE HWANG THAWAN KOOBURAT CS758 CLASS PROJECT FALL 2009 Extending Task-based Programming Model beyond Shared-memory Systems.
Practical techniques & Examples
A Dynamic World, what can Grids do for Multi-Core computing? Daniel Goodman, Anne Trefethen and Douglas Creager
NewsFlash!! Earth Simulator no longer #1. In slightly less earthshaking news… Homework #1 due date postponed to 10/11.
1 Chapter 1 Why Parallel Computing? An Introduction to Parallel Programming Peter Pacheco.
Starting Parallel Algorithm Design David Monismith Based on notes from Introduction to Parallel Programming 2 nd Edition by Grama, Gupta, Karypis, and.
Master/Slave Architecture Pattern Source: Pattern-Oriented Software Architecture, Vol. 1, Buschmann, et al.
Parallel Programming Models and Paradigms Prof. Rajkumar Buyya Cloud Computing and Distributed Systems (CLOUDS) Lab. The University of Melbourne, Australia.
Development of Parallel Simulator for Wireless WCDMA Network Hong Zhang Communication lab of HUT.
Optimus: A Dynamic Rewriting Framework for Data-Parallel Execution Plans Qifa Ke, Michael Isard, Yuan Yu Microsoft Research Silicon Valley EuroSys 2013.
March 18, 2008SSE Meeting 1 Mary Hall Dept. of Computer Science and Information Sciences Institute Multicore Chips and Parallel Programming.
CISC October Goals for today: Foster’s parallel algorithm design –Partitioning –Task dependency graph Granularity Concurrency Collective communication.
Reference: Message Passing Fundamentals.
DISTRIBUTED AND HIGH-PERFORMANCE COMPUTING CHAPTER 7: SHARED MEMORY PARALLEL PROGRAMMING.
Scripting Languages For Virtual Worlds. Outline Necessary Features Classes, Prototypes, and Mixins Static vs. Dynamic Typing Concurrency Versioning Distribution.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn.
CISC 879 : Software Support for Multicore Architectures John Cavazos Dept of Computer & Information Sciences University of Delaware
Mapping Techniques for Load Balancing
Lecture 2 – MapReduce CPE 458 – Parallel Programming, Spring 2009 Except as otherwise noted, the content of this presentation is licensed under the Creative.
Lecture 1 – Parallel Programming Primer CPE 458 – Parallel Programming, Spring 2009 Except as otherwise noted, the content of this presentation is licensed.
ADLB Update Recent and Current Adventures with the Asynchronous Dynamic Load Balancing Library Rusty Lusk Mathematics and Computer Science Division Argonne.
Introduction to Parallel Programming MapReduce Except where otherwise noted all portions of this work are Copyright (c) 2007 Google and are licensed under.
Reference: / Parallel Programming Paradigm Yeni Herdiyeni Dept of Computer Science, IPB.
Lecture 29 Fall 2006 Lecture 29: Parallel Programming Overview.
Lecture 4: Parallel Programming Models. Parallel Programming Models Parallel Programming Models: Data parallelism / Task parallelism Explicit parallelism.
An approach for solving the Helmholtz Equation on heterogeneous platforms An approach for solving the Helmholtz Equation on heterogeneous platforms G.
ICOM 5995: Performance Instrumentation and Visualization for High Performance Computer Systems Lecture 7 October 16, 2002 Nayda G. Santiago.
Parallel Programming Models Jihad El-Sana These slides are based on the book: Introduction to Parallel Computing, Blaise Barney, Lawrence Livermore National.
CCA Common Component Architecture Manoj Krishnan Pacific Northwest National Laboratory MCMD Programming and Implementation Issues.
1 678 Topics Covered (1) Part A: Foundation Socket Programming Thread Programming Elements of Parallel Computing Part B: Cluster Computing Elements of.
Chapter 3 Parallel Algorithm Design. Outline Task/channel model Task/channel model Algorithm design methodology Algorithm design methodology Case studies.
Introduction, background, jargon Jakub Yaghob. Literature T.G.Mattson, B.A.Sanders, B.L.Massingill: Patterns for Parallel Programming, Addison- Wesley,
Advanced / Other Programming Models Sathish Vadhiyar.
4.2.1 Programming Models Technology drivers – Node count, scale of parallelism within the node – Heterogeneity – Complex memory hierarchies – Failure rates.
Chapter 3 Parallel Programming Models. Abstraction Machine Level – Looks at hardware, OS, buffers Architectural models – Looks at interconnection network,
MATRIX MULTIPLY WITH DRYAD B649 Course Project Introduction.
并行程序设计基础. 2  并行程序设计概述  Approaches for Parallel Programs  Parallel Programming Model  Parallel Programming Paradigm.
Lecture 4 TTH 03:30AM-04:45PM Dr. Jianjun Hu CSCE569 Parallel Computing University of South Carolina Department of.
CS 484 Designing Parallel Algorithms Designing a parallel algorithm is not easy. There is no recipe or magical ingredient Except creativity We can benefit.
1 "Workshop 31: Developing a Hands-on Undergraduate Parallel Programming Course with Pattern Programming SIGCSE The 44 th ACM Technical Symposium.
Lecture 3 : Performance of Parallel Programs Courtesy : MIT Prof. Amarasinghe and Dr. Rabbah’s course note.
CS- 492 : Distributed system & Parallel Processing Lecture 7: Sun: 15/5/1435 Foundations of designing parallel algorithms and shared memory models Lecturer/
A Pattern Language for Parallel Programming Beverly Sanders University of Florida.
Motivation: dynamic apps Rocket center applications: –exhibit irregular structure, dynamic behavior, and need adaptive control strategies. Geometries are.
Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster www-unix.mcs.anl.gov/dbpp.
3/12/2013Computer Engg, IIT(BHU)1 OpenMP-1. OpenMP is a portable, multiprocessing API for shared memory computers OpenMP is not a “language” Instead,
NGS computation services: APIs and.
Parallel Computing Presented by Justin Reschke
Compilation of XSLT into Dataflow Graphs for Web Service Composition Peter Kelly Paul Coddington Andrew Wendelborn.
Holding slide prior to starting show. Processing Scientific Applications in the JINI-Based OGSA-Compliant Grid Yan Huang.
First INFN International School on Architectures, tools and methodologies for developing efficient large scale scientific computing applications Ce.U.B.
COMP7330/7336 Advanced Parallel and Distributed Computing MapReduce - Introduction Dr. Xiao Qin Auburn University
Department of Computer Science, Johns Hopkins University Lecture 7 Finding Concurrency EN /420 Instructor: Randal Burns 26 February 2014.
Introduction to Parallel Computing: MPI, OpenMP and Hybrid Programming
Lecture 1 – Parallel Programming Primer
Conception of parallel algorithms
Parallel Programming By J. H. Wang May 2, 2017.
Pattern Parallel Programming
Computer Engg, IIT(BHU)
The University of Adelaide, School of Computer Science
Parallel Algorithm Design
Parallel Programming in C with MPI and OpenMP
شیوه های موازی سازی parallelization methods
Hybrid Programming with OpenMP and MPI
Parallel Algorithm Models
Mattan Erez The University of Texas at Austin
Parallel Programming in C with MPI and OpenMP
MapReduce: Simplified Data Processing on Large Clusters
Mattan Erez The University of Texas at Austin
Presentation transcript:

Parallel Programming Models and Paradigms An Introduction WW Grid Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab. The University of Melbourne Melbourne, Australia www.gridbus.org

Presentation Outline Introduction Parallel Application Development Strategies Code Granularity and Levels of Parallelism Parallel Modes and Tools Methodical Design Steps Parallelization Paradigms Summary

Parallel Programming is a Complex Task The development of parallel applications largely dependent on the availability of adequate software tools and environments. Parallel software developers handle problems/issues/challenges such as: Non-determinism, communication, synchronization, data partitioning and distribution, load-balancing, fault-tolerance, heterogeneity, shared or distributed memory, deadlocks, and race conditions.

Users’ Expectations from Parallel Programming Environments

Approaches for Parallel Programs Development Implicit Parallelism Supported by parallel languages and parallelizing compilers that take care of identifying parallelism, the scheduling of calculations and the placement of data. Explicit Parallelism In this approach, the programmer is responsible for most of the parallelization effort such as task decomposition, mapping task to processors, the communication structure. This approach is based on the assumption that the user is often the best judge of how parallelism can be exploited for a particular application.

Strategies for Development

Parallel Programming Models and Tools Shared Memory Model DSM Threads/OpenMP (enabled for clusters) Java threads (HKU JESSICA, IBM cJVM) Message Passing Model PVM MPI Hybrid Model Mixing shared and distributed memory model Using OpenMP and MPI together Object and Service Oriented Models Wide area distributed computing technologies OO: CORBA, DCOM, etc. Services: Web Services-based service composition 14

Levels of Parallelism PVM/MPI Threads Compilers CPU Task i-l Task i Code-Granularity Code Item Large grain (task level) Program Medium grain (control level) Function (thread) Fine grain (data level) Loop (Compiler) Very fine grain (multiple issue) With hardware PVM/MPI Task i-l Task i Task i+1 func1 ( ) { .... } func2 ( ) { .... } func3 ( ) { .... } Threads Compilers a ( 0 ) =.. b ( 0 ) =.. a ( 1 )=.. b ( 1 )=.. a ( 2 )=.. b ( 2 )=.. CPU + x Load

Responsible for Parallelisation

Methodical Design or Stages of Parallel Programs Partitioning Decomposition of computational activities and the data into small tasks – there exist number of paradigms – e.g. master worker, pipeline, divide and conquer, SPMD, and speculation. Communication Flow of information and coordination among tasks that are created in the portioning stage. Agglomeration Tasks and communication structure created in the above stages are evaluated for performance and implementation cost. Tasks may be grouped into larger tasks to improve communication. Individual communications can be bundled. Mapping / Scheduling Assigning tasks to processors such that job completion time is minimized and resource utilization is maximized. Even the cost of computation can be minimized based on QoS requirements.

Parallelisation Paradigms Task-Farming/Master-Worker Model Single-Program Multiple-Data (SPMD) Pipelining Divide and Conquer Speculation Parametric Computation Model Nimrod-G - for computational intensive parameteric applications. Gridbus Broker – for distributed data intensive parameteric applications.

Master Worker/Slave Model Master decomposes the problem into small tasks, distributes to workers and gathers partial results to produce final result Mapping/Load Balancing Static Dynamic When number of tasks are larger than the number of CPUs / they are know at runtime / CPUs are heterogeneous. Static

Single-Program Multiple-Data Most commonly used model. Each process executes the same piece of code, but on different parts of the data.—splitting the data among the available processors. Different names: geometric/domain decomposition, data parallelism.

Data Pipelining Suitable for fine grained parallelism. Also suitable for application involving multiple stages of execution, but need to operate on large number of data sets.

Divide and Conquer A problem is divided into two or more sub problems, and each of these sub problems are solved independently, and their results are combined. 3 operations: split, compute, and join. Master-worker/task-farming is like divide and conquer with master doing both split and join operation.

Speculative Parallelism It used when it is quite difficult to achieve parallelism through one of the previous paradigms. Studying Grid computing/Java instead of C++ course increases the likelihood/chances of getting a job is an example of speculation! Problems with complex dependencies – use look ahead execution. Employing different algorithms for the same problem—the first one to give the final solution id the one that is chosen.

Summary Parallel programming is complex, but exciting as it appears similar to a real world working model.: Parallel App Dev strategies: automatic, library-based, and explicit parallelisation. Levels: fine grain, loop, function, and task Modes: shared, distributed, message passing, hybrid, parametric. Methodical design: partitioning, communication, agglomeration, and mapping. Parallelization Paradigms: Master worker/task farming, SPMD, data pipelining, divide and conquer, and speculations.

Reference L. Silva and R. Buyya, Parallel Programming Models and Paradigms, High Performance Cluster Computing: Programming and Applications, Rajkumar Buyya (editor), ISBN 0-13-013785-5, Prentice Hall PTR, NJ, USA, 1999.