1. Lecture 17. Vector Machine, and Intel MMX/SSEx Extensions
COSC3330 Computer Architecture
Lecture 17. Vector Machine, and
Intel MMX/SSEx Extensions
Instructor: Weidong Shi (Larry), PhD
Computer Science Department
University of Houston

2. Topics
Vector Machine
Intel MMX/SSEx Extensions

3. Supercomputer
A supercomputer is a hardware and software system that provides close to the maximum performance that can currently be achieved.
CDC6600 (Cray, 1964) regarded as first supercomputer
First RISC processor !
Seymour Cray, Father of Supercomputing, main designer

4. History of Supercomputers
In 70s-80s, Supercomputer  Vector Machine

5. Vector Supercomputers
Epitomized by Cray-1, 1976:
One of best known & successful supercomputer
Installed at LANL for $8.8 million
Vector Extension
Vector Registers
Vector Instructions
Implementation
Highly Pipelined Functional Units
Interleaved Memory System
No Data Caches
No Virtual Memory
Programming Cray-1
FORTRAN
Auto vectorizing compiler!

6. Vector Processing
Vector processors have high-level operations that work on linear arrays of numbers: "vectors"
SCALAR
(1 operation)
VECTOR
(N operations) v1 v2 v3 +
vector
length + r1 r2 r3
add r3, r1, r2
add.vv v3, v1, v2

7. SIMD
SIMD (single instruction multiple data) architecture performs the same operation on multiple data elements in parallel
PADDW MM0, MM1

8. Vector Length Register
Vector Register
Scalar Registers
Vector Registers
v15
r15 v0 r0
[0]
[1]
[2]
[VLRMAX-1]
Vector Length Register
VLR

9. Vector Arithmetic Instructions
ADDV v3, v1, v2 v1 + + + + + + v2 v3
[0]
[1]
[VLR-1]

10. Vector Load and Store Instructions
Vector Load/Store
Vector Load and Store Instructions
LV v1, r1, r2
Vector Register v1
Memory
Base, r1
Stride, r2

11. Vector Code Example # C code for (i=0; i<64; i++)
C[i] = A[i] + B[i];
# Scalar Code
LI R4, 64
loop:
L.D F0, 0(R1)
L.D F2, 0(R2)
ADD.D F4, F2, F0
S.D F4, 0(R3)
DADDIU R1, 8
DADDIU R2, 8
DADDIU R3, 8
DSUBIU R4, 1
BNEZ R4, loop
# Vector Code
LI VLR, 64
LV V1, R1
LV V2, R2
ADDV.D V3, V1, V2
SV V3, R3

12. VLIW: Very Long Instruction Word
Int Op 2
Mem Op 1
Mem Op 2
FP Op 1
FP Op 2
Int Op 1
Two Integer Units,
Single Cycle Latency
Two Load/Store Units,
Three Cycle Latency
Two Floating-Point Units,
Four Cycle Latency
Multiple operations packed into one instruction
Each operation slot is for a fixed function
Constant operation latencies are specified
Architecture requires guarantee of:
Parallelism within an instruction => no x-operation RAW check
No data use before data ready => no data interlocks

13. Vector Instruction Set Advantages
Compact
one short instruction encodes N operations
Expressive, tells hardware that these N operations:
are independent
use the same functional unit
access disjoint registers
access registers in same pattern as previous instructions
access a contiguous block of memory (unit-stride load/store)
Scalable
can run same code on more parallel pipelines (lanes)

14. Vector Arithmetic Execution
Use deep pipeline (=> fast clock) to execute element operations
Simplifies control of deep pipeline because elements in vector are independent (=> no hazards!) V1 V2 V3
Six stage multiply pipeline

15. Vector Instruction Execution
ADDV C,A,B
C[1]
C[2]
C[0]
A[3]
B[3]
A[4]
B[4]
A[5]
B[5]
A[6]
B[6]
Execution using one pipelined functional unit
C[4]
C[8]
C[0]
A[12]
B[12]
A[16]
B[16]
A[20]
B[20]
A[24]
B[24]
C[5]
C[9]
C[1]
A[13]
B[13]
A[17]
B[17]
A[21]
B[21]
A[25]
B[25]
C[6]
C[10]
C[2]
A[14]
B[14]
A[18]
B[18]
A[22]
B[22]
A[26]
B[26]
C[7]
C[11]
C[3]
A[15]
B[15]
A[19]
B[19]
A[23]
B[23]
A[27]
B[27]
Execution using four pipelined functional units

16. Vector Memory System 1 2 3 4 5 6 7 8 9 A B C D E F +
Cray-1, 16 banks, 4 cycle bank busy time, 12 cycle latency
Bank busy time: Cycles between accesses to same bank 1 2 3 4 5 6 7 8 9 A B C D E F +
Base
Stride
Vector Registers
Memory Banks
Address Generator

17. Vector Unit Structure Functional Unit Vector Registers Lane
Elements 0, 4, 8, …
Elements 1, 5, 9, …
Elements 2, 6, 10, …
Elements 3, 7, 11, …
Memory Subsystem

18. Vector Instruction Parallelism
Can overlap execution of multiple vector instructions
example machine has 32 elements per vector register and 8 lanes
Load Unit
Multiply Unit
Add Unit
load
mul
add
time
load
mul
add
Instruction issue
Complete 24 operations/cycle while issuing 1 short instruction/cycle

19. Vector Chaining Vector version of register bypassing
introduced with Cray-1
Memory V1
Load Unit
Mult. V2 V3
Chain
Add V4 V5
Chain
LV v1
MULV v3,v1,v2
ADDV v5, v3, v4

20. Vector Chaining Advantage
Load
Mul
Add
Time
Without chaining, must wait for last element of result to be written before starting dependent instruction
With chaining, can start dependent instruction as soon as first result appears
Load
Mul
Add

21. Automatic Code Vectorization
for (i=0; i < N; i++)
C[i] = A[i] + B[i];
load
add
store
Iter. 1
Iter. 2
Scalar Sequential Code
Vector Instruction
load
add
store
Iter. 1
Iter. 2
Vectorized Code
Time
Vectorization is a massive compile-time reordering of operation sequencing
 requires extensive loop dependence analysis

22. Vector Scatter/Gather
Want to vectorize loops with indirect accesses:
for (i=0; i<N; i++)
A[i] = B[i] + C[D[i]]
Indexed load instruction (Gather)
LV vD, rD # Load indices in D vector
LVI vC, rC, vD # Load indirect from rC base
LV vB, rB # Load B vector
ADDV.D vA,vB,vC # Do add
SV vA, rA # Store result rD rC vD vC vB vA 1 3 5 2 11 30 +

23. Vector Scatter/Gather
Scatter example:
for (i=0; i<N; i++)
A[B[i]]++;
Is following a correct translation?
LV vB, rB # Load indices in B vector
LVI vA, rA, vB # Gather initial A values
ADDV vA, vA, 1 # Increment
SVI vA, rA, vB # Scatter incremented values rB rA vB vA 1 3 5 2 11 30 +1

24. Vector Conditional Execution
Problem: Want to vectorize loops with conditional code:
for (i=0; i<N; i++)
if (A[i]>0) then
A[i] = B[i];
Solution: Add vector mask (or flag) registers
1 bit per element …and maskable vector instructions
vector operation becomes NOP at elements where mask bit is clear
Code example:
CVM # Turn on all elements
LV vA, rA # Load entire A vector
SGTVS.D vA, F0 # Set bits in mask register where A>0
LV vA, rB # Load B vector into A under mask
SV vA, rA # Store A back to memory under mask

25. Masked Vector Instructions
B[3]
A[4]
B[4]
A[5]
B[5]
A[6]
B[6]
M[3]=0
M[4]=1
M[5]=1
M[6]=0
M[2]=0
M[1]=1
M[0]=0
Write data port
Write Enable
A[7]
B[7]
M[7]=1
Simple Implementation
execute all N operations, turn off result writeback according to mask
C[4]
C[5]
C[1]
Write data port
A[7]
B[7]
M[3]=0
M[4]=1
M[5]=1
M[6]=0
M[2]=0
M[1]=1
M[0]=0
M[7]=1
Density-Time Implementation
scan mask vector and only execute elements with non-zero masks

26. Intel MMX/SSE/SSE2 Extension Why MMX?
Accelerate multimedia and communications applications.
Maintain full compatibility with existing operating systems and applications.
Exploit inherent parallelism in multimedia and communication algorithms
Includes new instructions and data types to improve performance.

27. First Step: Examine Code
Examined a wide range of applications: graphics, MPEG video, music synthesis, speech compression, speech recognition, image processing, games, video conferencing.
Identified and analyzed the most compute-intensive routines

28. Common Characteristics
Small integer data types: e.g. 8-bit pixels, 16-bit audio samples
Small, highly repetitive loops
Frequent multiply-and-accumulate
Compute-intensive algorithms
Highly parallel operations

29. IA-32 SIMD Development
MMX (Multimedia Extension) was introduced in 1996 (Pentium with MMX and Pentium II).
SSE (Streaming SIMD Extension) was introduced with Pentium III.
SSE2 was introduced with Pentium 4.
SSE3 was introduced with Pentium 4 supporting hyper-threading technology. SSE3 adds 13 more instructions.
SSE4 was introduced in SSE4 added 54 instructions.

30. MMX Data Types

31. MMX
Packed integer types allow operations to be applied on multiple integers 31

32. Aliases to Existing FP registers:79
NaN or infinity as real
because bits are
ones.
11…11
MMX is hidden behind FPU
MMX and FPU can not be used at the same time.
Big overhead to switch
MM0~MM7

33. MMX Instructions
57 MMX instructions are defined to perform the parallel operations on multiple data elements packed into 64-bit data types.
Basic arithmetic: add, subtract, multiply, arithmetic shift and multiply-add
Comparison
Conversion: pack & unpack
Logical
Shift
Move: register-to-register
Load/Store: 64-bit and 32-bit
All instructions except for data move use MMX registers as operands.

34. Packed Add Word with wrap around
Each Addition is independent
Rightmost overflows and wraps around

35. Saturation
Saturation: if addition results in overflow or underflow, the result is clamped to the largest or smallest value representable.
This is important for pixel calculations where this would prevent a wrap-around add from causing a black pixel to suddenly turn white

36. No Mode There is no "saturation mode bit”:
a new mode bit would require a change to the operating system.
Separate instructions are used to generate wrap-around and saturating results.

37. Packed Add Word with unsigned saturation
Each Addition is independent
Rightmost saturates

38. Multiply-Accumulate
multiply-accumulate operations are fundamental to many signal processing algorithms like vector-dot-products, matrix multiplies, FIR and IIR Filters, FFTs, DCTs etc

39. Packed Multiply-Add
Multiply bytes generating four 32-bit results. Add the 2 products on the left for one result and the 2 products on the right for the other result.

40. Packed Parallel Compare
Result can be used as a mask to select elements from different inputs using a logical operation, eliminating branchs.

41. Conditional Select
The Chroma Keying example demonstrates how conditional selection using the MMX instruction set removes branch mis-predictions, in addition to performing multiple selection operations in parallel. Text overlay on a pix/video background, and sprite overlays in games are some of the other operations that would benefit from this technique.

42. Chroma Keying + =

43. Chroma Keying (con’t)
Take pixels from the picture with the airplane on a green background.
A compare instruction builds a mask for that data. That mask is a sequence of bytes that are all ones or all zeros.
We now know what is the unwanted background and what we want to keep.

44. Create Mask
Assume pixels alternate green/not_green

45. Combine: !AND, AND, OR

46. SSE SSE introduced eight 128-bit data registers (called XMM registers)46

47. SSE Programming Environment
XMM0 |
XMM7
MM0 |
MM7
EAX, EBX, ECX, EDX
EBP, ESI, EDI, ESP

48. SSE Data Type SSE extensions introduced one new data type
128-Bit Packed Single-Precision Floating-Point Data Type
SSE 2 introduced five data types 48

49. Inline Assembly Code
Assembly language source code that is inserted directly into a HLL program.
Compilers such as Microsoft Visual C++ and GCC have compiler-specific directives that identify inline ASM code.
Simple to code because there are no external names, memory models, or naming conventions involved.
Decidedly not portable because it is written for a single platform.

50. __asm directive in Microsoft Visual C++
void X_aligned_memcpy_sse2(void* dest, const void* src, const unsigned long size_t) {
__asm
mov esi, src; //src pointer
mov edi, dest; //dest pointer
mov ebx, size_t; //ebx is our counter
shr ebx, 7; //divide by 128 (8 * 128bit registers)
loop_copy: …
movdqa xmm0, 0[ESI]; //move data from src to registers
movdqa xmm1, 16[ESI];
movdqa xmm2, 32[ESI];
movntdq 0[EDI], xmm0; //move data from registers to dest
movntdq 16[EDI], xmm1;
movntdq 32[EDI], xmm2;
jnz loop_copy; //loop please
loop_copy_end: }
Mark the beginning of a block of assembly language statements

51. Intel MMX/SSE Intrinsics
Intrinsics are C/C++ functions and procedures for MMX/SSE instructions
With instrinsics, one can program using these instructions indirectly using the provided intrinsics
In general, there is a one-to-one correspondence between MMX/SSE instructions and intrinsics
_mm_<opcode>_<suffix>
ps: packed single-precision
ss: scalar single-precision 51

52. Intrinsics #include <xmmintrin.h> __m128 a , b , c;
c = _mm_add_ps( a , b );
float a[4] , b[4] , c[4];
for( int i = 0 ; i < 4 ; ++ i )
    c[i] = a[i] + b[i];
// a = b * c + d / e;
__m128 a = _mm_add_ps( _mm_mul_ps( b , c ) ,
_mm_div_ps( d , e ) );