1. CS 286 Computer Organization and Architecture
Parallel Computers
Organizations and Architecture
Department of Computer Science
Southern Illinois University Edwardsville
Fall, 2018
Dr. Hiroshi Fujinoki

2. CS 286 – Parallel Computers
Four hardware architecture for “parallel computers”
 Tightly-Coupled Multi-Processor System
 Functionally-Specialized Multi-Processor System
 Loosely-Coupled Multi-Processor System
 Distributed Systems (“most loosely coupled systems”)
Mult_Sched/001

3. CS 286 – Parallel Computers
 Tightly-Coupled Multi-Processor System
Multi-Processor System (multi-processor motherboard)
Single-Processor System with a multi-core processor
Motherboard
Motherboard
Processor
Processor
Processor Core
(ALU and others)
Multi-Processor
System
Single-Processor System
with multi-core processor
Mult_Sched/002

4. CS 286 – Parallel Computers
 Tightly-Coupled Multi-Processor System
Multi-Processor System (multi-processor motherboard)
Two processors on a motherboard
Mult_Sched/003

5. CS 286 – Parallel Computers
 Tightly-Coupled Multi-Processor System
Single-Processor System with a multi-core processor
CPU cores
Mult_Sched/004

6. CS 286 – Parallel Computers
 Functionally-Specialized Multi-Processor System
Examples:
GPU on graphics card
(GPU = “Graphic Processing Unit”)
Built-in processor on high-speed disk controllers or NICs
(especially those using DMA)
 Processor sends graphic command
to GPU 
Graphic Interface
 GPU processes image data in the
graphic-card memory 
Motherboard
 Graphic-card performs D/A
conversion using DAC. 
 Graphic-card sends analog image
signals (RGB-signals) to monitor 
DAC
Monitor
(CRT, Flat Panel)
Video RAM (“VRAM”)
GPU
Processor
Mult_Sched/005

7. CS 286 – Parallel Computers
 Functionally-Specialized Multi-Processor System
Examples:
GPU on graphics card
(GPU = “Graphic Processing Unit”)
Control Program (in ROM)
DMA SCSI I/O card
CPU
Mult_Sched/006

8. CS 286 – Parallel Computers
 Loosely-Coupled Multi-Processor System
Multi-Systemboard (multiple motherboard) computers
A computer with multiple
motherboards (“blades”)
Processor
Blades communicate
through the bus
Each blade is a computer
Communication delay
over the bus
System Board
(Motherboard)
at least “s” order
Memory
“Bus”
Computer System
Mult_Sched/007

9. CS 286 – Parallel Computers
 Loosely-Coupled Multi-Processor System
Multi-Systemboard (multiple motherboard) computers
Mult_Sched/008

10. CS 286 – Parallel Computers
 Distributed Systems (“most loosely coupled systems”)
Processor
Local Memory
Secondary Storage
Other I/O
File (data)
Data Migration
Process
(executable codes)
Process Migration
AS 1
AS 4
AS 2
AS 3
Processor
Local Memory
Secondary Storage
Other I/O
Network
Mult_Sched/009

11. CS 286 – Parallel Computers
Three different types of tightly-coupled multi-processor systems
(1) “Fine-grained” multi-processor parallel computers
(2) “Medium-grained” multi-processor parallel computers
(3) “Coarse-grained” multi-processor parallel computers
Mult_Sched/010

12. CS 286 – Parallel Computers
Fine-Grained Multi-Process
Granularity: 20~100 instructions
Fine-grained = instruction-level multi-processing
A = B + C;
X = Y + Z;
W = A + X;
Your program
(binary executable)
Dependency
CPU
synchronization
Mult_Sched/011

13. CS 286 – Parallel Computers
Medium-Grained Multi-Process
Granularity: threads (100~10,000 instructions/thread)
Medium-grained = thread-level multi-processing
Your program
(binary executable)
ThreadA
ThreadB
ThreadC
ThreadD
Processor
Processor
a thread of execution is the smallest sequence of programmed
instructions that can be managed independently by a scheduler,
which is typically a part of the operating system
Mult_Sched/012

14. CS 286 – Parallel Computers
Medium-Grained Multi-Process
Medium-grained = thread-level multi-processing
The number of threads
each program consists of
Mult_Sched/010

15. CS 286 – Parallel Computers
Medium-Grained Multi-Process
Granularity: threads (100~10,000 instructions/thread)
Example: Web Browser
ThreadA -- Display thread (text output & jpeg image processing)
ThreadB -- Taking user inputs (edit boxes, radio boxes in the browser window
ThreadC -- Network input (receiving data from network)
ThreadD -- Network output (sending data to network)
ThreadA ThreadB ThreadC ThreadD
Receiving
data
Displaying
data
User makes
inputs
Transmit
data
Receiving
data
Mult_Sched/013

16. CS 286 – Parallel Computers
Medium-Grained Multi-Process
Example: Web Browser
ThreadA -- Display thread (text output & jpeg image processing)
ThreadB -- Taking user inputs (edit boxes, radio boxes in the browser window
ThreadC -- Network input (receiving data from network)
ThreadD -- Network output (sending data to network)
ThreadA ThreadB ThreadC ThreadD
Receiving
data
Displaying
data
User makes
inputs
Transmit
data
Receiving
data
Browser execution
with better responses
Mult_Sched/014

17. CS 286 – Parallel Computers
Coarse-Grained Multi-Process
Coarse-grained = (process-level) multi-tasking
Process assignment to multiple processors in multi-tasking environment
Memory
Processor
Time
Mult_Sched/015

18. CS 286 – Parallel Computers
Coarse-Grained Multi-Process
Memory
Processor
Mult_Sched/016

19. CS 286 – Parallel Computers
Coarse-Grained Multi-Process
Granularity: programs (1 ~ 50 threads/program)
Coarse-grained = process-level multi-tasking
Process assignment to multiple processors in multi-tasking environment
Processor Pool
Granularity = ms order
1ms 1GHz) = 1 million instructions
Memory
100ms 1GHz) = 100M instructions
Time
Mult_Sched/017

20. CS 286 – Parallel Computers
SIMD Vector Computer: Cray (multiple parallel processors on a mother board)
New_Technologies/019

21. CS 286 – Parallel Computers
New_Technologies/019
CS 286 – Parallel Computers
Vector Datapath Processors (super computers)
Representative processors in this generation
C-90 and Y-MP (Cray), VAX 9000 (Digital), … IF ID
EX1 ME WB
EX2 IF ID
EX1 ME WB
EX3
EX2
EX3
EX4
EX5 IF ID
EX1 ME WB
EX4
EX2
EX3
EX4
EX5
EX6 1 2 3 4 5 6 7 8 9 10
Clock Cycles

22. CS 286 – Parallel Computers
SIMD Vector Computer: Cray (multiple parallel processors on a mother board)
CRAY “Trinity”
New_Technologies/019

23. CS 286 – Parallel Computers
Cray (1975)
DELL’s “NOAA” (2017)
IBM “Summit” (2018)
New_Technologies/019