A Seminar on Optimizations for Modern Architectures Computer Science Seminar 2, Winter 2006 Lecturer: Erez Petrank

Topics today Administration Introduction Course topics Administration again

Course format Lectures 2-14 will be given by you. Each lecture will follow a lecture from a course by Ken Kennedy at Rice University. Slides “drafts” are in Original lectures took 80 minutes. 16 such lectures, one by each registered student.

Grading Presentation - at least 85%, Participation in class – at most 15%. Reduction of 5 points for each (unjustified) absence. Presentation: –Understanding the material –Slides effectiveness –Communicating the ideas

Seminar Requirement Easier than “standard”: –Material is taken from a book –Draft slides are available on the internet Harder than “standard”: –A presentation is 80 minutes instead of 50. –Presentation should be much better than draft slides. –Presentations may depend on previous material, so one actually has to listen… –Lecturer has to know what he is talking about. Goal: make people understand !

Scheduling Since lectures are not synch’ed with our 2*50 minutes schedule, one must be prepared to start after the previous talk ends. Lecturers should send me their talk slides by 12:30 the day of the talk. After the talk, please modify the slides to create a final version and send it to me by the end of Thursday.

The book Optimizing Compilers for Modern Architectures by Ken Kennedy & Randy Allen (Available at the library: 1 reserved copy at the library and 2 more for a week's loan. )

The nature of this topic Trying to automatically detect potential parallelization seems to be harder than expected. Requires some math, some common sense, some compiler knowledge, some NP-Hardness, etc. At the frontline of programming languages research today. Interesting both from research and from practice point of view.

An Example Can we run the following loop iterations in parallel? For (i=0, i<1000, i++) For (j=0, j<1000, j++) For (k=0, k<1000, k++) For (l=0, l<1000, l++) A(i+3,2j+2,k+5,l) = A(2j-i+k,2k,7,2i) A sufficient question is whether one iteration writes to a location in A that another iteration reads from… This translates to whether there exist two integer vectors (i 1,j 1,k 1,l 1 ) and (i 2,j 2,k 2,l 2 ) such that i 1 +3 = 2j 2 -i 2 +k 2 & 2j 1 +2 = 2k 2 & k 1 +5 = 7 & l 1 = 2i 2 (Solutions should also be within the loop bounds.)

An Example Do there exist two integer vectors (i 1,j 1,k 1,l 1 ) and (i 2,j 2,k 2,l 2 ) such that i 1 +3 = 2j 2 -i 2 +k 2 & 2j 1 +2 = 2k 2 & k 1 +5 = 7 & l 1 = 2i 2 ? This is a set of linear equations. Let’s solve them! Bad news: finding integer solutions for linear equations in NP-Hard. So what do we do? Restrict to special cases that are common and we have the math tools to solve…

Introduction to Course (Chapter 1) What kind of parallel platforms are we targeting? How “dependency” relates to all.

Moore’s Law

Features of Machine Architectures Pipelining. Superscalar: multiple execution units (pipelined). –When controlled by the software it is called VLIW. Vector operations Parallel processing –Shared memory, distributed memory, message-passing Registers Cache hierarchy Combinations of the above –Parallel-vector machines

Compiler’s Job Many advanced developers write platform- conscious code. –Parallelism, cache behavior, etc. Time consuming and bug prone. Bugs are extremely difficult to find. Holly grail: the compiler gets “standard” code and optimizes it to make best use of the hardware.

Main Tool: Reordering Changing the execution order of original code: –Do not wait for cache misses –Run instructions in parallel –Improve “branch prediction”, etc. Main question: Which orders preserve program semantics? –Dependence. –Safety

Dependnece Dependence is the main theme in this book. How do we determine dependence? How do we use this knowledge to improve programs? Next, we go through the various platform relevant properties: pipelining, vector instructions, superscalar, processor parallelism, memory hierarchy.

Earliest: Instruction Pipelining Instruction pipelining –DLX Instruction Pipeline Picture shows perfect pipelining, executing 1 inst/cycle Instruction fetch, Decode, execute, mem access, & write back to register. (RISC.)

Replicated Execution Logic Pipelined Execution Units Multiple Execution Units Typical programs do not exploit such pipelining or parallelism well. But a compiler can help by reordering instructions. Fine-grained parallelism

Parallelism Hazards Structural Hazard –A single bus brings instructions and data. So memory access cannot happen with instruction fetch. Data Hazard –The result produced by one instruction is required by the following instruction Control Hazard –Branches Hazards create stalls. Goal: minimize stalls.

Vector Instructions By the 70’s, the task of keeping the pipeline full had become cumbersome. Vector operations: an attempt to simplify instruction processing. Idea: long “vector” register, loaded, executed, or stored in a single operation. Example: »VLOAD V1,A »VLOADV2,B »VADDV3,V1,V2 »VSTOREV3,C

Vector Instructions The machine runs vector operations using the same pipelined hardware. Simplification: we know what is happening. Platform/system costs: processor has more registers, more instructions, cache complications, etc.

Vector Operations How does a programmer specify or how does a compiler identify vector operations? »DO I = 1, 64 »C(I) = A(I) + B(I) »D(I+1) = D(I) + C(I) »ENDDO

VLIW (Very Long Instruction Word) Multiple instruction issue on the same cycle –Wide word instruction Challenges for programmer/compiler –Find enough parallel instructions –Reschedule operations to minimize number of cycles. Avoid hazards –Dependence !

SMP Parallelism Multiple processors with uniform shared memory. Course grained (task) parallelism: independent tasks. Do not rely on one another for obtaining data or making control decisions.

Memory Hierarchy Memory speed falls behind processor speed, and gap still increasing. Solution: use a fast cache between memory and CPU. Implication: program cache behavior has a significant impact on program efficiency. Challenge: How can we enhance reuse? –Correct! We reorder instructions. Cache Mem CPU

Memory Hierarchy –Example: –DO I = 1, N – DO J = 1, M – C(I) = C(I) + A(J) –If M is small enough to let A[1:M] in the cache, then (almost) no cache misses on A. Otherwise, cache miss on A per loop iteration. –Solution for a cache of size L: –DO jj = 1, M, L – DO I = 1, N – DO J = jj, jj+L-1 – C(I) = C(I) + A(J)

Distributed Memory Memory packaged with processors –Message passing –Distributed shared memory SMP clusters –Shared memory on node, message passing off node Problem more complex: split computation and data. –Minimizing communication: data placement –Create copies? Maintain coherence, etc. We will probably not have time to discuss data partition in this seminar.

Some Words on Dependence Goal: aggressive transformations to improve performance Problem: when is a transformation legal? –Simple answer: when program meaning not changed. Meaning? –Same sequence of memory states? Too strong! –Same answers? Good, but in general undecidable. Need a sufficient condition. We use in this course: dependence. –Ensures instructions that access the same location (with at least one store) must not be reordered.

Bernstein Conditions When is it safe to run two instructions I1 and I2 in parallel? If none of the following holds: 1.I1 writes into a memory location that I2 reads 2.I2 writes into a memory location that I1 reads 3.Both I1 and I2 write to the same memory location Loop parallelism: Think of loop iterations as tasks

Bernstein Conditions Bernstein Conditions True Dependence: –I1 writes into a memory location that I2 reads –Do i=1,N –A(i+1) = A(i) + B(i) Bernstein Conditions Antidependence: –I2 writes into a memory location that I1 reads –Do i=1,N –A(i-1) = A(i) + B(i)

Bernstein Conditions Bernstein Conditions Output Dependence: –Both I1 and I2 write to the same memory location –Do i=1,N –S = A(i) + B(i) –Better: –X = 10 –X = 20

Compiler Technologies Program Transformations –Most architectural issues dealt with by reordering. Vectorization, parallelization, cache reuse enhancement –Challenges: Determining when transformations are legal Selecting transformations based on profitability All transformations require some understanding of the ways that instructions and statements depend on one another (share data).

A Common Problem: Matrix Multiplication DO I = 1, N DO J = 1, N C(J,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO This code will perform perfectly on a scalar machine. Compiler only needs to identify the common variable C(j,i) in the inner loop and put it in a register.

Pipelined Machine DO I = 1, N DO J = 1, N C(J,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO But Suppose it ran on a pipeline machine with 4 stages. Problem: inner (k) loop cannot be parallelized.

Problem for Pipelines Instructions in inner (k) loop are dependent. Solution: –work on several iterations of the J-loop simultaneously

MatMult for a Pipelined Machine DO I = 1, N, DO J = 1, N, 4 C(J,I) = 0.0 C(J+1,I) = 0.0 C(J+2,I) = 0.0 C(J+3,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) C(J+1,I) = C(J+1,I) + A(J+1,K) * B(K,I) C(J+2,I) = C(J+2,I) + A(J+2,K) * B(K,I) C(J+3,I) = C(J+3,I) + A(J+3,K) * B(K,I) ENDDO

Matrix Multiply on Vector Machines DO I = 1, N DO J = 1, N C(J,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO Reminder: this was the original code.

Problems for Vectors Inner loop must be vector –And should be stride 1 Vector registers have finite length (Cray: 64 elements) Solution –Strip mine the loop over the stride-one dimension to the register length (say, 64). –Move the iterate over strip loop to the innermost position

Vectorizing Matrix Multiply DO I = 1, N DO J = 1, N, 64 DO JJ = 0,63 C(JJ,I) = 0.0 ENDDO DO K = 1, N DO JJ = 0,63 C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO

MatMult for a Vector Machine DO I = 1, N DO J = 1, N, 64 C(J:J+63,I) = 0.0 DO K = 1, N C(J:J+63,I) = C(J:J+63,I) + A(J:J+63,K)*B(K,I) ENDDO

Matrix Multiply on Parallel SMPs DO I = 1, N DO J = 1, N C(J,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO On an SMP, we prefer to parallelize at a coarse level because of the cost involved in synchronizing the parallel runs.

Matrix Multiply on Parallel SMPs DO I = 1, N ! Independent for all I DO J = 1, N C(J,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO We can let each processor execute some of the outer loop iterations. (They should all have access to the full matrix.)

MatMult on a Shared-Memory MP PARALLEL DO I = 1, N DO J = 1, N C(J,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO END PARALLEL DO

MatMult on a Vector SMP PARALLEL DO I = 1, N DO J = 1, N, 64 C(J:J+63,I) = 0.0 DO K = 1, N C(J:J+63,I) = C(J:J+63,I) + A(J:J+63,K)*B(K,I) ENDDO

Matrix Multiply for Cache Reuse DO I = 1, N DO J = 1, N C(J,I) = 0.0 DO K = 1, N C(J,I) = C(J,I) + A(J,K) * B(K,I) ENDDO

Problems on Cache There is reuse of C but no reuse of A and B Solution –Block the loops so you get reuse of both A and B Multiply a block of A by a block of B and add to block of C

MatMult on a Uniprocessor with Cache DO I = 1, N, S DO J = 1, N, S DO p = I, I+S-1 DO q = J, J+S-1 C(q,p) = 0.0 ENDDO DO K = 1, N, S DO p = I, I+S-1 DO q = J, J+S-1 DO r = K, K+S-1 C(q,p) = C(q,p) + A(q,r) * B(r,p) ENDDO ST elements S 2 elements

MatMult on a Distributed-Memory MP PARALLEL DO I = 1, N PARALLEL DO J = 1, N C(J,I) = 0.0 ENDDO PARALLEL DO I = 1, N, S PARALLEL DO J = 1, N, S DO K = 1, N, T DO p = I, I+S-1 DO q = J, J+S-1 DO r = K, K+T-1 C(q,p) = C(q,p) + A(q,r) * B(r,p) ENDDO

Conclusion Tailoring the program to a given platform is difficult. Due to the speed of hardware change, modifications may be required repeatedly. Thus, it is useful to let the compiler do much of the work. How much can it do? This is the topic of this course.

Summary Modern computer architectures present many performance challenges Most of the problems can be overcome by changing order of instructions, and in particular, by transforming loop nests –Transformations are not obviously correct. Dependence tells us when this is feasible –Most of the course is concentrated on detection and use of dependence (or independence) for the various goals.

Administration Again

Registration Procedure 1.I’ll wait to get additional initial registration s for the next 2 hours. 2.I will send s asking students if they want to pick a lecture. 3.I will wait a limited time for responses and go for the next candidates.  If you want to ensure your place in this course, check your every 4 hours tomorrow and respond. 4.A schedule will be built and put on the internet with names of registered students. 5.(If you don’t get registered, you are still welcome as a listener…)