COMP60611 Fundamentals of Parallel and Distributed Systems Lecture 1 Overview of Parallel Computing John Gurd, Graham Riley Centre for Novel Computing School of Computer Science University of Manchester

Overview We first look at the whole process of developing (and maintaining) a computational system, based on an example (medium range weather forecasting) Application-oriented Levels of Abstraction Application Level Specification Level Algorithm Level Implementation Levels of Abstraction Program Level Computer Level Summary 16/02/2019

Correct results and good (high) performance: Background We are primarily concerned with: The design of applications for Execution on parallel and distributed computers which give Correct results and good (high) performance: These are concurrent systems Developing concurrent systems that do what they are supposed to do and which deliver high performance, where necessary, requires well-designed interactions between many different facets of computing, ranging from the applications themselves to the structure of the parallel computers which execute them. This interaction covers several Levels of Abstraction, which encompass fundamentally different views. The following five Levels of Abstraction demonstrate how these views gradually change from being application-oriented, at one end of the spectrum, to hardware-oriented, at the other. 16/02/2019

Levels of Abstraction The Application Level – here a relatively simple, and possibly informal, description of the application (i.e., the problem to be solved) is stated or developed. In the medium-range weather forecasting example, this description might be something like: “Once (or perhaps twice) daily, forecast the state of the global weather system for a period of ten days from now. For operational reasons, the computation on which this forecast is based must be achieved within an 8 hour 'window' each day.” 16/02/2019

Levels of Abstraction The Specification Level – here the simple application description is turned into a formal specification (abstract model) of the application problem, expressed in a suitable mathematical notation. For the weather prediction example, the global situation is modelled (roughly) by a small number of coupled partial differential equations (PDEs) representing conservation of momentum, energy and mass, and a state equation. These describe relationships between key continuous variables such as wind speed, density (including moisture content), temperature and pressure. 16/02/2019

Levels of Abstraction The Algorithm Level – here a systematic procedure for solving the problem is developed, based on some discrete data domain. In many cases, the problem itself cannot be solved directly, and so an approximation is used to solve a related problem. The discrete domains encountered here reflect the discrete nature of computer storage that will be encountered later during the development; nevertheless, they remain rather abstract entities. Parallelism arises in two basic forms: data-parallelism and task-parallelism. For the weather prediction example, the global continuum is represented by a grid-point approximation (i.e. using spot values of the key variables at each grid-point). Progress in time is approximated by a series of discrete time-steps, governed by finite difference equations derived from the PDEs – mostly data-parallelism over the finite difference grid. 16/02/2019

Levels of Abstraction The Program Level – here the algorithm is expressed as a program, moving the concerns further towards the restrictions associated with real computer storage. Unfortunately, programming languages tend to reflect the hardware architecture of the computer that will execute the compiled code, often leading to 'unnatural' constraints on the algorithm. A typical example for medium-range weather forecasting is the Met Office’s Unified Model (UM) code, which merges developments in both weather and climate modelling into a single meteorological FORTRAN program with (in the latest versions) message-passing parallel constructs in MPI (Message Passing Interface). The Integrated Forecasting System (IFS) of the European Centre for Medium-range Weather Forecasting (ECMWF) is another such example. The use of message-passing has forced the (parallel) algorithm in certain directions that may not have been followed if sequential hardware had continued to be used. The message-passing approach was adopted due to availability of specific parallel hardware. 16/02/2019

Levels of Abstraction The Computer Level – here the program is implemented on parallel digital hardware. The object-code generated by the compiler, or embedded in the runtime system, causes low-level electronic transactions across the data paths of the hardware, between processors, memory modules and peripheral devices. In the weather prediction example: in 2009, the Met Office bought a large (several thousand multicore processors) IBM Power 6 Cluster system which now runs the UM code in operational mode. The runtime message-passing system is the manufacturer's implementation of MPI. Generation of suitable parallel code has not been straightforward; it has taken many years, and involved a considerable amount of expert manpower. Tools have been of limited assistance, being rudimentary or overly ambitious. Such experiences abound in the practice of parallel computing; our later focus is on trying to understand why this is so. 16/02/2019

Numerical Weather Prediction Roughly speaking, 1/3rd of the 6-8 hour daily compute-time is pre-processing, to establish initial conditions at each grid-point, 1/3rd is time-stepping iterations, to compute successive new weather states, and 1/3rd is post-processing, to derive the interesting qualitative conditions (cloud, rainfall, etc.). Limiting factors are: accuracy of computed initial state; resolution of space and time; quality (accuracy and divergence) of time-stepping iterative approximation. The quality of a forecast can be measured post hoc by the length of time that (some of) its predictions remain within a given error of the actual weather. 16/02/2019

Numerical Weather Prediction In practice, parallelism of several thousand-fold is being used in this production system. Parallelism is exploited across the data 'spaces' (independent vertical columns of the atmosphere), and across the different forecasts of the EPS (Ensemble Prediction System). Attention is increasingly focussed on additional pre-processing (to improve the computed initial states) and probabilistic algorithms for predicting the 'most likely' outcome based on ensemble forecasts. 16/02/2019

Concurrent Systems: Issues Correct behaviour: In the sense of verification (“did we build it right?”), as opposed to validation (“did we build the right thing?”) Performance: What limits the performance of a system? What happens to performance as the number of processors/cores increases (scalability analysis) In this module, the focus is on the design phase: What options do we have in building a system and how do we choose between them? Why is developing a concurrent solution so much more difficult than developing a sequential solution? 16/02/2019

A Wider View… Concurrent systems occur in many applications beyond computer systems For example, airports, supermarkets, banking (e.g. ATM systems) These involve queueing networks and transaction processing Many of these systems contain complex computing systems, too (ATM networks, reservation systems and associated databases) Such systems do not necessarily compute a result, in the way that the weather prediction system does. Rather, they support some complex, on-going behaviour. Mobile systems add further levels of complexity due to their dynamic nature For example, mobile telecommunications systems, wireless access for mobile computing (PDAs and laptops) and Airport Traffic Control systems The fundamental (concurrency) issues remain the same. 16/02/2019

Summary There are several key aspects to concurrent systems Principally we will be concerned with correctness and performance Each aspect lends itself to a different treatment at the design stage Need to use the appropriate techniques in order to construct ‘good’ concurrent systems In this module we shall look at three techniques: Formal behaviour modelling (using the Promela language and SPIN model checking tool) Performance modelling (of algorithms) Queueing theory and discrete simulation We shall also introduce basic concepts for mobile systems 16/02/2019