Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 2 Shared-Memory Model and Threads Objectives At the end of this module you should be able to: Describe the shared-memory model of parallel programming Describe the differences between the fork/join model and the general threads model Demonstrate how to implement domain and functional decompositions using threads Decide whether a variable in a multithreaded program should be shared or private

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 3 Shared-Memory Model and Threads Cooperating Parallel Processes Parallel computing multiple processes working together to speed the solution of a task Working together process cooperation Kinds of cooperation Sharing information (communication) Keeping out of each others way (synchronization)

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 4 Shared-Memory Model and Threads The Shared-Memory Model Shared Memory CPU Private Memory CPU Private Memory CPU Private Memory

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 5 Shared-Memory Model and Threads Evaluating Parallel Models How do processes share information? How do processes synchronize? In shared-memory model, both accomplished through shared variables Communication: buffer Synchronization: semaphore

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 6 Shared-Memory Model and Threads Methodology Study problem, sequential program, or code segment Look for opportunities for parallelism Use threads to express parallelism

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 7 Shared-Memory Model and Threads What Is a Process? A program in some state of execution Code Data Logical address space Information about a process includes Process state Program counter Values of CPU registers Memory management information

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 8 Shared-Memory Model and Threads What Is a Thread? A unit of control within a process Carver and Tai Main thread executes programs main function Main thread may create other threads to execute other functions Threads have own program counter, copy of CPU registers, and stack of activation records Threads share processs data, code, address space, and other resources Threads have lower overhead than processes

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 9 Shared-Memory Model and Threads Another Way of Looking at It So why does this view matter? Thread 0 (Program Counter, Registers, etc.) Address Space File Descriptors Other Data Process

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 10 Shared-Memory Model and Threads Why This View Matters Thread 0 (Program Counter, Registers, etc.) Address Space File Descriptors Other Data Process Thread 1 (Program Counter, Registers, etc.)

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 11 Shared-Memory Model and Threads What Are Threads Good For? Making programs easier to understand Overlapping computation and I/O Improving responsiveness of GUIs Improving performance through parallel execution

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 12 Shared-Memory Model and Threads Fork/Join Programming Model When program begins execution, only master thread active Master thread executes sequential portions of program For parallel portions of program, master thread forks (creates or awakens) additional threads At join (end of parallel section of code), extra threads are suspended or die

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 13 Shared-Memory Model and Threads Master Thread ForkJoin Fork Other Threads

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 14 Shared-Memory Model and Threads Relating Fork/Join to Code for { } } Sequential code Parallel code Sequential code Parallel code Sequential code

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 15 Shared-Memory Model and Threads More General Threads Model When program begins execution, only one user thread, called the main thread, is active The main thread can create other threads, which execute other functions Created threads can also create additional threads How this is done varies according to programming language or API

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 16 Shared-Memory Model and Threads Fork/Join versus General Model You can implement fork/join in the general model Hence fork/join a special case of general model More structured More easily optimized General model More flexible Better support for functional decompositions

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 17 Shared-Memory Model and Threads Incremental Parallelization Sequential program a special case of threaded program Programmers can add parallelism incrementally Profile program execution Repeat Choose best opportunity for parallelization Transform sequential code into parallel code Until further improvements not worth the effort

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 18 Shared-Memory Model and Threads Utility of Threads Threads are flexible enough to implement Domain decomposition Functional decomposition Pipelining

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 19 Shared-Memory Model and Threads Domain Decomposition Using Threads Shared Memory Thread 0Thread 2 Thread 1 f ( )

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 20 Shared-Memory Model and Threads Shared Memory Functional Decomposition Using Threads Thread 0Thread 1 e ( ) g ( )h ( ) f ( )

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 21 Shared-Memory Model and Threads Pipelining Using Threads Thread 0Thread 2Thread 1 Shared Memory InputOutput e ( )f ( )g ( ) Data set 2 Data sets 5, 6,... Data set 4 Data set 3 Data set 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 22 Shared-Memory Model and Threads Shared versus Private Variables Shared Variables Private Variables Private Variables Thread

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 23 Shared-Memory Model and Threads Domain Decomposition Sequential Code: int a[1000], i; for (i = 0; i < 1000; i++) a[i] = foo(i);

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 24 Shared-Memory Model and Threads Domain Decomposition Sequential Code: int a[1000], i; for (i = 0; i < 1000; i++) a[i] = foo(i); Thread 0: for (i = 0; i < 500; i++) a[i] = foo(i); Thread 1: for (i = 500; i < 1000; i++) a[i] = foo(i);

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 25 Shared-Memory Model and Threads Domain Decomposition Sequential Code: int a[1000], i; for (i = 0; i < 1000; i++) a[i] = foo(i); Thread 0: for (i = 0; i < 500; i++) a[i] = foo(i); Thread 1: for (i = 500; i < 1000; i++) a[i] = foo(i); Shared Private

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 26 Shared-Memory Model and Threads Functional Decomposition int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k);... } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; }

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 27 Shared-Memory Model and Threads Functional Decomposition int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } Thread 0 Thread 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 28 Shared-Memory Model and Threads Functional Decomposition volatile int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } Thread 0 Thread 1 Static variable: Shared

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 29 Shared-Memory Model and Threads Functional Decomposition volatile int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } Thread 0 Thread 1 Heap variable: Shared

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 30 Shared-Memory Model and Threads Functional Decomposition volatile int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x. k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } Thread 0 Thread 1 Functions local variables: Private

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 31 Shared-Memory Model and Threads Shared and Private Variables Shared variables Static variables Heap variables Contents of run-time stack at time of call Private variables Loop index variables Run-time stack of functions invoked by thread

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 32 Shared-Memory Model and Threads The Shared-Memory Model (Reprise) Shared Memory CPU Private Memory CPU Private Memory CPU Private Memory

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 33 Shared-Memory Model and Threads The Threads Model Shared Variables Thread Private Variables Thread Private Variables Thread Private Variables

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 34 Shared-Memory Model and Threads References Jim Beveridge and Robert Wiener, Multithreading Applications in Win32®, Addison-Wesley (1997). David R. Butenhof, Programming with POSIX® Threads, Addison-Wesley, (1997). Richard H. Carver and Kuo-Chung Tai, Modern Multithreading: Implementing, Testing, and Debugging Java and C++/Pthreads/ Win32 Programs, Wiley-Interscience (2006). Doug Lea, A Java Fork/Join Framework, Proceedings of the ACM 2000 Conference on Java Grande, pp Michael J. Quinn, Parallel Programming in C with MPI and OpenMP, McGraw-Hill (2004).

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 35 Shared-Memory Model and Threads