1. IT 251 Computer Organization and Architecture
Optimization and Parallelism
Modified by R. Helps from original notes by Chia-Chi Teng

2. Pipelining vs. Parallel processing
In both cases, multiple “things” processed by multiple “functional units”
Pipelining: each thing is broken into a sequence of pieces, where each piece is handled by a different (specialized) functional unit
Parallel processing: each thing is processed entirely by a single functional unit
We will briefly introduce the key ideas behind parallel processing
instruction level parallelism
thread-level parallelism

3. Paralleism depends on scale
Multiple computers working in concert
Modern supercomputers
“Beowulf” architecture
Multiple network paths
Chunk size is a complete program module
Capstone project to use idle cycle on campus (install virtual machine)
home and offspring
Parallelism within a computer: Multiple cores
Chunk size is a subroutine or small app
Parallelism within a computer: single core
Multiple execution units within core
Chunk size is on the instruction level
Parallelism in GPU
Optimized for many similar math operations
Linear algebra and DSP
Chunk size is a math operation

4. Exploiting Parallelism
Of the computing problems for which performance is important, many have inherent parallelism
Best example: computer games
Graphics, physics, sound, AI etc. can be done separately
Furthermore, there is often parallelism within each of these:
Each pixel on the screen’s color can be computed independently
Non-contacting objects can be updated/simulated independently
Artificial intelligence of non-human entities done independently
Another example: Google queries
Every query is independent
Google search is read-only!!
Read-only means no synchronization problems

5. Parallelism at the Instruction Level
add eax <- eax, edx
or eax <- eax, ebx
mov edx <- 12(esp)
addi edx <- edx, eax
sub ecx <- ecx, 1
Dependences?
RAW
WAW
WAR
pipeline stalled
When can we reorder instructions?
When should we reorder instructions?
Solution:
Multiple instructions executing in parallel at *same* stage
add eax <- eax, edx
or eax <- eax, ebx
mov edx <- 12(esp)
sub ecx <- ecx, 1
addi edx <- edx, eax

6. Modern CPU Design Instruction Control Execution Fetch Control Address
Cache
Retirement
Unit
Register
File
Instruction
Decode
Instructions
Operations
Register Updates
Prediction OK?
Execution
Functional
Units
Note separate execution units for integer and floating point.
Also note that in some modern processors there are multiple parallel units for some operations (see textbook ch 5)
Integer/
Branch
General
Integer FP
Add FP
Mult/Div
Load
Store
Operation Results
Addr.
Addr.
Data
Data
Data
Cache

7. Superscalar Processor
Definition: A superscalar processor can issue and execute multiple instructions in one cycle. The instructions are retrieved from a sequential instruction stream and are usually scheduled dynamically.
Benefit: without programming effort, superscalar processor can take advantage of instruction level parallelism in most programs.
Most CPUs since about 1998 are superscalar.
Intel: since Pentium Pro

8. Superscaler Hardware

9. Multicycle FP Operations
Multiple functional units: one approach
Special-purpose hardware for FP operations
Increased latency causes more frequent stalls
Single cycle integer unit
Fully pipelined multiplier F D M W
Fully pipelined FP adder
Non-pipelined divider
can cause “structural” hazards

10. Dynamic scheduling Out-of-order execution engine: one view (Pentium 4)
Fetching,
decoding,
translation of x86
instrs to uops
to support precise exceptions and
recovery from mispredicted branches
Image from com

11. Exploiting Parallelism at the Data Level
Consider adding together two arrays (linear algebra):
void
array_add(int A[], int B[], int C[], int length) {
int i;
for (i = 0 ; i < length ; ++ i) {
C[i] = A[i] + B[i]; }
Operating on one element at a time +

12. Exploiting Parallelism at the Data Level
Consider adding together two arrays:
void
array_add(int A[], int B[], int C[], int length) {
int i;
for (i = 0 ; i < length ; ++ i) {
C[i] = A[i] + B[i]; }
Operating on one element at a time +

13. Exploiting Parallelism at the Data Level (SIMD)
Consider adding together two arrays:
void
array_add(int A[], int B[], int C[], int length) {
int i;
for (i = 0 ; i < length ; ++ i) {
C[i] = A[i] + B[i]; }
Operate on MULTIPLE elements + + + +
Single Instruction,
Multiple Data (SIMD)

14. Intel SSE/SSE2 as an example of SIMD
Added new 128 bit registers (XMM0 – XMM7), each can store
4 single precision FP values (SSE) 4 * 32b
2 double precision FP values (SSE2) 2 * 64b
16 byte values (SSE2) 16 * 8b
8 word values (SSE2) 8 * 16b
4 double word values (SSE2) 4 * 32b
bit integer value (SSE2) 1 * 128b
Perform four ADDs in parallel
4.0 (32 bits) +
3.5 (32 bits)
-2.0 (32 bits)
2.3 (32 bits)
1.7 (32 bits)
2.0 (32 bits)
-1.5 (32 bits)
0.3 (32 bits)
5.2 (32 bits)
6.0 (32 bits)
2.5 (32 bits)
SIMD Single Instruction Multiple Data
SSE = Stream SIMD Extensions (Intel 1999)

15. Is it always that easy? Not always… a more challenging example:
unsigned
sum_array(unsigned *array, int length) {
int total = 0;
for (int i = 0 ; i < length ; ++ i) {
total += array[i]; }
return total;
Is there parallelism here?

16. We first need to restructure the code
unsigned
sum_array2(unsigned *array, int length) {
unsigned total, i;
unsigned temp[4] = {0, 0, 0, 0};
for (i = 0 ; i < length & ~0x3 ; i += 4) {
temp[0] += array[i];
temp[1] += array[i+1];
temp[2] += array[i+2];
temp[3] += array[i+3]; }
total = temp[0] + temp[1] + temp[2] + temp[3];
for ( ; i < length ; ++ i) {
total += array[i];
return total;
Length & ~0x3 = mask off lower two bits (count by fours)
All the temp I’s add in parallel
Last loop is for array leftovers < 3

17. Then we can write SIMD code for the hot part
unsigned
sum_array2(unsigned *array, int length) {
unsigned total, i;
unsigned temp[4] = {0, 0, 0, 0};
for (i = 0 ; i < length & ~0x3 ; i += 4) {
temp[0] += array[i];
temp[1] += array[i+1];
temp[2] += array[i+2];
temp[3] += array[i+3]; }
total = temp[0] + temp[1] + temp[2] + temp[3];
for ( ; i < length ; ++ i) {
total += array[i];
return total;
What if there are 8 processors available?

18. Thread level parallelism: Multi-Core Processors
Two (or more) complete processors, fabricated on the same silicon chip
Execute instructions from two (or more) programs/threads at same time #1 #2
IBM Power5

19. Multi-Cores are Everywhere
Intel Core i7: 2, 4 or 6 x86 processors on same chip
XBox360: 3 PowerPC cores
Sony Playstation 3: Cell processor, an asymmetric multi-core with 9 cores (1 general-purpose, 8 special purpose SIMD processors)
Exercise: How many parallel processors in a GPU card?

20. Why Multi-cores Now?
Number of transistors we can put on a chip growing exponentially…

21. … and performance growing too…
But power is growing even faster!!
Power has become limiting factor in current chips

22. As programmers, do we care?
What happens if we run a program on a multi-core?
void
array_add(int A[], int B[], int C[], int length) {
int i;
for (i = 0 ; i < length ; ++i) {
C[i] = A[i] + B[i]; }
Can’t easily split on instruction level between cores #1 #2

23. What if we want a program to run on both processors?
We have to explicitly tell the machine exactly how to do this
This is called parallel programming or concurrent programming
There are many parallel/concurrent programming models
We will look at a relatively simple one: fork-join parallelism
Posix threads and explicit synchronization

24. Review: Amdahl’s Law
Execution time after improvement =
Time affected by improvement +
Time unaffected by improvement
Amount of improvement
Example: Many computing tasks are I/O-bound, and the speed of the input and output devices limits the overall system performance.
Improved CPU performance alone has a limited effect on overall system speed.

25. Fork/Join Logical Example
Fork N-1 threads
Break work into N pieces (and do it)
Join (N-1) threads
void
array_add(int A[], int B[], int C[], int length) {
cpu_num = fork(N-1);
int i;
for (i = cpu_num ; i < length ; i += N) {
C[i] = A[i] + B[i]; }
join();
Fork() creates a separate thread, which an run on a separate core
Simplified code for fork()
How good is this with caches?

26. How does this help performance?
Parallel speedup measures improvement from parallelization:
time for best serial version
time for version with p processors
What can we realistically expect?
speedup(p) =

27. Reason #1: Amdahl’s Law
In general, the whole computation is not (easily) parallelizable
Serial regions

28. Reason #1: Amdahl’s Law New Execution Time = 1-s + s P
Suppose a program takes 1 unit of time to execute serially
A fraction of the program, s, is inherently serial (unparallelizable)
For example, consider a program that, when executing on one processor, spends 10% of its time in a non-parallelizable region. How much faster will this program run on a 3-processor system?
What if you had 8 cores?
What is the maximum speedup from parallelization?
New Execution Time =
1-s + s P
Class Exercise
.3T + .1T = 0.4T, speedup old_T / new_T = 1/0.4 = 2.5x
Theoretically best speedup for perfectly parallelizable = N times for N processors
But what about overhead?
New Execution Time =
.9T +
.1T 3
Speedup =

29. Reason #2: Overhead Forking and joining is not instantaneous
void
array_add(int A[], int B[], int C[], int length) {
cpu_num = fork(N-1);
int i;
for (i = cpu_num ; i < length ; i += N) {
C[i] = A[i] + B[i]; }
join();
Forking and joining is not instantaneous
Involves communicating between processors
May involve calls into the operating system
Depends on the implementation
New Execution Time =
1-s + s
overhead(P) P

30. Programming Explicit Thread-level Parallelism
As noted previously, the programmer must specify how to parallelize
But, want path of least effort
Division of labor between the Human and the Compiler
Humans: good at expressing parallelism, bad at bookkeeping
Compilers: bad at finding parallelism, good at bookkeeping
Want a way to take serial code and say “Do this in parallel!” without:
Having to manage the synchronization between processors
Having to know a priori how many processors the system has
Deciding exactly which processor does what
Replicate the private state of each thread
OpenMP: an industry standard set of compiler extensions
Works very well for programs with structured parallelism.
Visual Studio with Concurrency Runtime

31. New Programming Model
Scalable
Task Oriented
Statelessness

32. Performance Optimization
Until you are an expert, first write a working version of the program
Then, and only then, begin tuning, first collecting data, and iterate
Otherwise, you will likely optimize what doesn’t matter
“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.” -- Sir Tony Hoare
Discuss Pareto chart
Expect this in exam

33. Profiling
Profiling is data collection for recording the performance of each part of the program, so that it can be optimised.
Requires special tools
Tools
Visual Studio: MSDN tools link
GCC: use GPROF or similar (link)
Other tools exist for other environments

34. Fix only the big problems
Pareto analysis to identify most significant problem
Remember Amdahl’s law
Pareto diagram from Wikipedia

35. Conventional Wisdom (CW) in Computer Architecture
Old CW: Power is free, but transistors expensive
New CW: Power Wall - Power expensive, transistors “free”
Can’t put more transistors on a chip than have power to turn on
Old CW: Multiplies slow, but loads fast
New CW: Memory Wall - Loads slow, multiplies fast
200 clocks to DRAM, but even FP multiplies only 4 clocks
Old CW: More ILP (instruction level parallelism) via compiler / architecture innovation
Branch prediction, speculation, Out-of-order execution, VLIW, …
New CW: ILP Wall - Diminishing returns on more ILP
Old CW: 2X CPU Performance every 18 months
New CW: Power Wall+MemoryWall+ILP Wall = Brick Wall
The doubling of uniprocessor perf will take longer
Power wall Turbo mode in i7 works by turning off mall but one cores for power reasons

36. Conventional Wisdom
“The promise of parallelism has fascinated researchers for at least three decades. In the past, parallel computing efforts have shown promise and gathered investment, but in the end uniprocessor computing always prevailed. Nevertheless, we argue general-purpose computing is taking an irreversible step toward parallel architectures.”
– Berkeley View, December 2006

37. Parallelism again? What’s different this time?
“This shift toward increasing parallelism is not a triumphant stride forward based on breakthroughs in novel software and architectures for parallelism; instead, this plunge into parallelism is actually a retreat from even greater challenges that thwart efficient silicon implementation of traditional uniprocessor architectures.”
– Berkeley View, December 2006
HW/SW Industry bet its future that breakthroughs will appear before it’s too late
view.eecs.berkeley.edu

38. Number of Cores/Socket
We need revolution, not evolution
Software or architecture alone can’t fix parallel programming problem, need innovations in both
“Multicore” 2X cores per generation: 2, 4, 8, …
“Manycore” 100s is highest performance per unit area, and per Watt, then 2X per generation: 64, 128, 256, 512, 1024 …
Multicore architectures & Programming Models good for 2 to 32 cores won’t evolve to Manycore systems of 1000’s of processors Desperately need HW/SW models that work for Manycore or will run out of steam (as ILP ran out of steam at 4 instructions)

39. Morgan Kaufmann Publishers
14 February, 2018
FIGURE 7.17 Four recent multiprocessors, each using two sockets for the processors. Starting from the upper left
hand corner, the computers are: (a) Intel Xeon e5345 (Clovertown), (b) AMD Opteron X (Barcelona), (c) Sun UltraSPARC T (Niagara 2), and (d) IBM Cell QS20. Note that the Intel Xeon e5345 (Clovertown) has a separate north bridge chip not found in the other microprocessors. Copyright © 2009 Elsevier, Inc. All rights reserved.
Chapter 7 — Multicores, Multiprocessors, and Clusters

40. Morgan Kaufmann Publishers
14 February, 2018
FIGURE 7.21 Performance of LBMHD on the four multicores. Copyright © 2009 Elsevier, Inc. All rights reserved.
Chapter 7 — Multicores, Multiprocessors, and Clusters

42. Summary Multi-core is having more than one processor on the same chip.
Most PCs/servers and game consoles are already multi-core
Results from Moore’s law and power constraint
Exploiting multi-core requires parallel programming
Automatically extracting parallelism too hard for compiler, in general.
But, can have compiler do much of the bookkeeping for us
OpenMP
Fork-Join model of parallelism
At parallel region, fork a bunch of threads, do the work in parallel, and then join, continuing with just one thread
Expect a speedup of less than P on P processors
Amdahl’s Law: speedup limited by serial portion of program
Overhead: forking and joining are not free
Everything is changing
Old conventional wisdom is out
We desperately need new approach to HW and SW based on parallelism since industry has bet its future that parallelism work

43. Administrivia Course rating - BYU Course outcome survey – blackboard
Professionalism
Lab 10 & HW 11, due tomorrow
Final exam, available right after class, due 5PM next Monday, Dec 14th
Cover everything in Block 2
Pipeline
x86 stack structure
Memory structure, cache, VM
Interrupt, IO, disk
Late work deadline, next Monday, Dec 14th
Quiz 8