1. COMP60611 Fundamentals of Parallel and Distributed Systems
Lecture 1
Overview of Parallel Computing
John Gurd, Graham Riley
Centre for Novel Computing
School of Computer Science
University of Manchester

2. Overview
We first look at the whole process of developing (and maintaining) a computational system, based on an example (medium range weather forecasting)
Application-oriented Levels of Abstraction
Application Level
Specification Level
Algorithm Level
Implementation Levels of Abstraction
Program Level
Computer Level
Summary
16/02/2019

3. Correct results and good (high) performance:
Background
We are primarily concerned with:
The design of applications for
Execution on parallel and distributed computers which give
Correct results and good (high) performance:
These are concurrent systems
Developing concurrent systems that do what they are supposed to do and which deliver high performance, where necessary, requires well-designed interactions between many different facets of computing, ranging from the applications themselves to the structure of the parallel computers which execute them.
This interaction covers several Levels of Abstraction, which encompass fundamentally different views. The following five Levels of Abstraction demonstrate how these views gradually change from being application-oriented, at one end of the spectrum, to hardware-oriented, at the other.
16/02/2019

4. Levels of Abstraction
The Application Level – here a relatively simple, and possibly informal, description of the application (i.e., the problem to be solved) is stated or developed.
In the medium-range weather forecasting example, this description might be something like:
“Once (or perhaps twice) daily, forecast the state of the global weather system for a period of ten days from now. For operational reasons, the computation on which this forecast is based must be achieved within an 8 hour 'window' each day.”
16/02/2019

5. Levels of Abstraction
The Specification Level – here the simple application description is turned into a formal specification (abstract model) of the application problem, expressed in a suitable mathematical notation.
For the weather prediction example, the global situation is modelled (roughly) by a small number of coupled partial differential equations (PDEs) representing conservation of momentum, energy and mass, and a state equation. These describe relationships between key continuous variables such as wind speed, density (including moisture content), temperature and pressure.
16/02/2019

6. Levels of Abstraction
The Algorithm Level – here a systematic procedure for solving the problem is developed, based on some discrete data domain. In many cases, the problem itself cannot be solved directly, and so an approximation is used to solve a related problem. The discrete domains encountered here reflect the discrete nature of computer storage that will be encountered later during the development; nevertheless, they remain rather abstract entities. Parallelism arises in two basic forms: data-parallelism and task-parallelism.
For the weather prediction example, the global continuum is represented by a grid-point approximation (i.e. using spot values of the key variables at each grid-point). Progress in time is approximated by a series of discrete time-steps, governed by finite difference equations derived from the PDEs – mostly data-parallelism over the finite difference grid.
16/02/2019

7. Levels of Abstraction
The Program Level – here the algorithm is expressed as a program, moving the concerns further towards the restrictions associated with real computer storage. Unfortunately, programming languages tend to reflect the hardware architecture of the computer that will execute the compiled code, often leading to 'unnatural' constraints on the algorithm.
A typical example for medium-range weather forecasting is the Met Office’s Unified Model (UM) code, which merges developments in both weather and climate modelling into a single meteorological FORTRAN program with (in the latest versions) message-passing parallel constructs in MPI (Message Passing Interface). The Integrated Forecasting System (IFS) of the European Centre for Medium-range Weather Forecasting (ECMWF) is another such example.
The use of message-passing has forced the (parallel) algorithm in certain directions that may not have been followed if sequential hardware had continued to be used. The message-passing approach was adopted due to availability of specific parallel hardware.
16/02/2019

8. Levels of Abstraction
The Computer Level – here the program is implemented on parallel digital hardware. The object-code generated by the compiler, or embedded in the runtime system, causes low-level electronic transactions across the data paths of the hardware, between processors, memory modules and peripheral devices.
In the weather prediction example: in 2009, the Met Office bought a large (several thousand multicore processors) IBM Power 6 Cluster system which now runs the UM code in operational mode. The runtime message-passing system is the manufacturer's implementation of MPI.
Generation of suitable parallel code has not been straightforward; it has taken many years, and involved a considerable amount of expert manpower. Tools have been of limited assistance, being rudimentary or overly ambitious. Such experiences abound in the practice of parallel computing; our later focus is on trying to understand why this is so.
16/02/2019

9. Numerical Weather Prediction
Roughly speaking, 1/3rd of the 6-8 hour daily compute-time is pre-processing, to establish initial conditions at each grid-point, 1/3rd is time-stepping iterations, to compute successive new weather states, and 1/3rd is post-processing, to derive the interesting qualitative conditions (cloud, rainfall, etc.).
Limiting factors are:
accuracy of computed initial state;
resolution of space and time;
quality (accuracy and divergence) of time-stepping iterative approximation.
The quality of a forecast can be measured post hoc by the length of time that (some of) its predictions remain within a given error of the actual weather.
16/02/2019

10. Numerical Weather Prediction
In practice, parallelism of several thousand-fold is being used in this production system. Parallelism is exploited across the data 'spaces' (independent vertical columns of the atmosphere), and across the different forecasts of the EPS (Ensemble Prediction System).
Attention is increasingly focussed on additional pre-processing (to improve the computed initial states) and probabilistic algorithms for predicting the 'most likely' outcome based on ensemble forecasts.
16/02/2019

11. Concurrent Systems: Issues
Correct behaviour:
In the sense of verification (“did we build it right?”), as opposed to validation (“did we build the right thing?”)
Performance:
What limits the performance of a system?
What happens to performance as the number of processors/cores increases (scalability analysis)
In this module, the focus is on the design phase:
What options do we have in building a system and how do we choose between them?
Why is developing a concurrent solution so much more difficult than developing a sequential solution?
16/02/2019

12. A Wider View…
Concurrent systems occur in many applications beyond computer systems
For example, airports, supermarkets, banking (e.g. ATM systems)
These involve queueing networks and transaction processing
Many of these systems contain complex computing systems, too (ATM networks, reservation systems and associated databases)
Such systems do not necessarily compute a result, in the way that the weather prediction system does. Rather, they support some complex, on-going behaviour.
Mobile systems add further levels of complexity due to their dynamic nature
For example, mobile telecommunications systems, wireless access for mobile computing (PDAs and laptops) and Airport Traffic Control systems
The fundamental (concurrency) issues remain the same.
16/02/2019

13. Summary There are several key aspects to concurrent systems
Principally we will be concerned with correctness and performance
Each aspect lends itself to a different treatment at the design stage
Need to use the appropriate techniques in order to construct ‘good’ concurrent systems
In this module we shall look at three techniques:
Formal behaviour modelling (using the Promela language and SPIN model checking tool)
Performance modelling (of algorithms)
Queueing theory and discrete simulation
We shall also introduce basic concepts for mobile systems
16/02/2019