CS 286 Computer Organization and Architecture Parallel Computers Organizations and Architecture Department of Computer Science Southern Illinois University Edwardsville Fall, 2018 Dr. Hiroshi Fujinoki E-mail: hfujino@siue.edu

CS 286 – Parallel Computers Four hardware architecture for “parallel computers”  Tightly-Coupled Multi-Processor System  Functionally-Specialized Multi-Processor System  Loosely-Coupled Multi-Processor System  Distributed Systems (“most loosely coupled systems”) Mult_Sched/001

CS 286 – Parallel Computers  Tightly-Coupled Multi-Processor System Multi-Processor System (multi-processor motherboard) Single-Processor System with a multi-core processor Motherboard Motherboard Processor Processor Processor Core (ALU and others) Multi-Processor System Single-Processor System with multi-core processor Mult_Sched/002

CS 286 – Parallel Computers  Tightly-Coupled Multi-Processor System Multi-Processor System (multi-processor motherboard) Two processors on a motherboard Mult_Sched/003

CS 286 – Parallel Computers  Tightly-Coupled Multi-Processor System Single-Processor System with a multi-core processor CPU cores Mult_Sched/004

CS 286 – Parallel Computers  Functionally-Specialized Multi-Processor System Examples: GPU on graphics card (GPU = “Graphic Processing Unit”) Built-in processor on high-speed disk controllers or NICs (especially those using DMA)  Processor sends graphic command to GPU  Graphic Interface  GPU processes image data in the graphic-card memory  Motherboard  Graphic-card performs D/A conversion using DAC.   Graphic-card sends analog image signals (RGB-signals) to monitor  DAC Monitor (CRT, Flat Panel) Video RAM (“VRAM”) GPU Processor Mult_Sched/005

CS 286 – Parallel Computers  Functionally-Specialized Multi-Processor System Examples: GPU on graphics card (GPU = “Graphic Processing Unit”) Control Program (in ROM) DMA SCSI I/O card CPU Mult_Sched/006

CS 286 – Parallel Computers  Loosely-Coupled Multi-Processor System Multi-Systemboard (multiple motherboard) computers A computer with multiple motherboards (“blades”) Processor Blades communicate through the bus Each blade is a computer Communication delay over the bus System Board (Motherboard) at least “s” order Memory “Bus” Computer System Mult_Sched/007

CS 286 – Parallel Computers  Loosely-Coupled Multi-Processor System Multi-Systemboard (multiple motherboard) computers Mult_Sched/008

CS 286 – Parallel Computers  Distributed Systems (“most loosely coupled systems”) Processor Local Memory Secondary Storage Other I/O File (data) Data Migration Process (executable codes) Process Migration AS 1 AS 4 AS 2 AS 3 Processor Local Memory Secondary Storage Other I/O Network Mult_Sched/009

CS 286 – Parallel Computers Three different types of tightly-coupled multi-processor systems (1) “Fine-grained” multi-processor parallel computers (2) “Medium-grained” multi-processor parallel computers (3) “Coarse-grained” multi-processor parallel computers Mult_Sched/010

CS 286 – Parallel Computers Fine-Grained Multi-Process Granularity: 20~100 instructions Fine-grained = instruction-level multi-processing A = B + C; X = Y + Z; W = A + X; Your program (binary executable) Dependency CPU synchronization Mult_Sched/011

CS 286 – Parallel Computers Medium-Grained Multi-Process Granularity: threads (100~10,000 instructions/thread) Medium-grained = thread-level multi-processing Your program (binary executable) ThreadA ThreadB ThreadC ThreadD Processor Processor a thread of execution is the smallest sequence of programmed instructions that can be managed independently by a scheduler, which is typically a part of the operating system Mult_Sched/012

CS 286 – Parallel Computers Medium-Grained Multi-Process Medium-grained = thread-level multi-processing The number of threads each program consists of Mult_Sched/010

CS 286 – Parallel Computers Medium-Grained Multi-Process Granularity: threads (100~10,000 instructions/thread) Example: Web Browser ThreadA -- Display thread (text output & jpeg image processing) ThreadB -- Taking user inputs (edit boxes, radio boxes in the browser window ThreadC -- Network input (receiving data from network) ThreadD -- Network output (sending data to network) ThreadA ThreadB ThreadC ThreadD Receiving data Displaying data User makes inputs Transmit data Receiving data Mult_Sched/013

CS 286 – Parallel Computers Medium-Grained Multi-Process Example: Web Browser ThreadA -- Display thread (text output & jpeg image processing) ThreadB -- Taking user inputs (edit boxes, radio boxes in the browser window ThreadC -- Network input (receiving data from network) ThreadD -- Network output (sending data to network) ThreadA ThreadB ThreadC ThreadD Receiving data Displaying data User makes inputs Transmit data Receiving data Browser execution with better responses Mult_Sched/014

CS 286 – Parallel Computers Coarse-Grained Multi-Process Coarse-grained = (process-level) multi-tasking Process assignment to multiple processors in multi-tasking environment Memory Processor Time Mult_Sched/015

CS 286 – Parallel Computers Coarse-Grained Multi-Process Memory Processor Mult_Sched/016

CS 286 – Parallel Computers Coarse-Grained Multi-Process Granularity: programs (1 ~ 50 threads/program) Coarse-grained = process-level multi-tasking Process assignment to multiple processors in multi-tasking environment Processor Pool Granularity = ms order 1ms (@ 1GHz) = 1 million instructions Memory 100ms (@ 1GHz) = 100M instructions Time Mult_Sched/017

CS 286 – Parallel Computers SIMD Vector Computer: Cray (multiple parallel processors on a mother board) New_Technologies/019

CS 286 – Parallel Computers New_Technologies/019 CS 286 – Parallel Computers Vector Datapath Processors (super computers) Representative processors in this generation C-90 and Y-MP (Cray), VAX 9000 (Digital), … IF ID EX1 ME WB EX2 IF ID EX1 ME WB EX3 EX2 EX3 EX4 EX5 IF ID EX1 ME WB EX4 EX2 EX3 EX4 EX5 EX6 1 2 3 4 5 6 7 8 9 10 Clock Cycles

CS 286 – Parallel Computers SIMD Vector Computer: Cray (multiple parallel processors on a mother board) CRAY “Trinity” New_Technologies/019

CS 286 – Parallel Computers Cray (1975) DELL’s “NOAA” (2017) IBM “Summit” (2018) New_Technologies/019