INTEL CONFIDENTIAL Shared Memory Considerations Introduction to Parallel Programming – Part 4

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Review & Objectives Previously: Described method to identify opportunities for parallelism in code segments and applications At the end of this part you should be able to: Describe the shared-memory model of parallel programming Demonstrate how to implement domain and task decompositions using threads Decide whether a variable in a multithreaded program should be shared or private 2

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Cooperating Parallel Processes Parallel computing  multiple processes working together to speed the solution of a task Working together  process cooperation Kinds of cooperation Sharing information (communication) Keeping out of each other’s way (synchronization) 3

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. The Shared-Memory Model 4 Shared Memory Core Private Memory Core Private Memory Core Private Memory

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Evaluating Parallel Models How do processes share information? How do processes synchronize? In shared-memory model, both accomplished through shared variables Communication: buffer Synchronization: semaphore 5

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Methodology Study problem, sequential program, or code segment Look for opportunities for parallelism Use threads to express parallelism 6

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. What Is a Process? A program in some state of execution Code Data Logical address space Information about a process includes Process state Program counter Values of core’s registers Memory management information 7

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. What Is a Thread? “A unit of control within a process” — Carver and Tai Main thread executes program’s “main” function Main thread may create other threads to execute other functions Threads have own program counter, copy of core registers, and stack of activation records Threads share process’s data, code, address space, and other resources Threads have lower overhead than processes 8

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Utility of Threads Threads are flexible enough to implement Domain decomposition Task decomposition Pipelining 9

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Domain Decomposition Using Threads 10 Shared Memory Thread 0Thread 2 Thread 1 f ( )

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Task Decomposition Using Threads 11 Shared Memory Thread 0Thread 1 e ( ) g ( )h ( ) f ( )

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Pipelining Using Threads 12 Thread 0Thread 2Thread 1 Shared Memory InputOutput e ( )f ( )g ( ) Data set 2 Data sets 5, 6,... Data set 4 Data set 3 Data set 1

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Shared versus Private Variables 13 Shared Variables Private Variables Private Variables Thread

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Domain Decomposition Sequential Code: int a[1000], i; for (i = 0; i < 1000; i++) a[i] = foo(i); 14

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Domain Decomposition Sequential Code: int a[1000], i; for (i = 0; i < 1000; i++) a[i] = foo(i); Thread 0: for (i = 0; i < 500; i++) a[i] = foo(i); Thread 1: for (i = 500; i < 1000; i++) a[i] = foo(i); 15

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Domain Decomposition Sequential Code: int a[1000], i; for (i = 0; i < 1000; i++) a[i] = foo(i); Thread 0: for (i = 0; i < 500; i++) a[i] = foo(i); Thread 1: for (i = 500; i < 1000; i++) a[i] = foo(i); 16 Shared Private

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Task Decomposition int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k);... } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } 17

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Task Decomposition int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } 18 Thread 0 Thread 1

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Task Decomposition Volatile int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } 19 Thread 0 Thread 1 Static variable: Shared

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Task Decomposition Volatile int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } 20 Thread 0 Thread 1 Heap variable: Shared

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Task Decomposition Volatile int e; main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); } int f(int *x, int k) { int a; a = e * x[k] * x[k]; return a; } int g(int *x, int k) { int a; k = k-1; a = e / x[k]; return a; } 21 Thread 0 Thread 1 Function’s local variables: Private

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Shared and Private Variables Shared variables Static variables Heap variables Contents of run-time stack at time of call Private variables Loop index variables Run-time stack of functions invoked by thread 22

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. The Shared-Memory Model (Reprise) 23 Shared Memory Core Private Memory Core Private Memory Core Private Memory

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. The Threads Model 24 Shared Variables Thread Private Variables Thread Private Variables Thread Private Variables

Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. References Jim Beveridge and Robert Wiener, Multithreading Applications in Win32®, Addison-Wesley (1997). David R. Butenhof, Programming with POSIX® Threads, Addison-Wesley, (1997). Richard H. Carver and Kuo-Chung Tai, Modern Multithreading: Implementing, Testing, and Debugging Java and C++/Pthreads/ Win32 Programs, Wiley-Interscience (2006). Michael J. Quinn, Parallel Programming in C with MPI and OpenMP, McGraw-Hill (2004). 25