Understanding The Nehalem Core Note: The examples herein are mostly illustrative. They have shortcommings compared to the real implementation in favour of easier visibility.

April Agenda Pipelines Branch Prediction Superscalar execution Out-Of-Order execution

April Understanding CPUs Optimizing performance is pretty much tied to understanding what a CPU actually does. VTune, Thread Profiler and other tools are mere helpers in order to understand the operation of the CPU. The following gives some illustrative ideas on how the most important concepts of modern CPU design work. Not in any case this corresponds to the actual implementation, but rather outlines some basic ideas.

April Architecture Block Diagram Disclaimer: This block diagram is for example purposes only. Significant hardware blocks have been arranged or omitted for clarity. Branch Target Buffer Microcode Sequencer Register Allocation Table (RAT) 32 KB Instruction Cache Next IP Instruction Decode (4 issue) Fetch / Decode Retire Re-Order Buffer (ROB) – 128 entry IA Register Set To L2 Cache/Memory Port Bus Unit Reservation Stations (RS) 32 entry Scheduler / Dispatch Ports 32 KB Data Cache Execute Port FP Add SIMD Integer Arithmetic Memory Order Buffer (MOB) Load Store Addr FP Div/Mul Integer Shift/Rotate SIMD Integer Arithmetic Port Store Data This is too complicated for the beginning! The best way to understand it is to construct the CPU yourself

April Our CPU Construction CPU Data Let‘s assume nothing for the moment – The CPU is just a black box with data coming in and data going out

April Modern Processor Technology In order to make use of modern CPUs some concepts must be clear 1.Pipelines 2.Prediction 3.Superscalarity 4.Out-Of-Order Execution 5.Vectorization We will go step by step through these concepts and make contact to indicators for sub-optimal performance on the Nehalem architecture

April Pipelines - What a CPU really understands … A CPU can distinguish between different concepts Instructions – what to do with the arguments Arguments - Usually registers that contain numbers or addresses A register can be thought of as a piece of memory directly in the CPU. It is as big as the architecture is wide (64bit). Specialized registers can be wider (SSE registers: 128bit). The ALU can only operate on registers!

April Pipelines - What a CPU really understands … d 63 db 4a 8d 04 9f 4f 8d 1c 9a 0f 28 c ff f ed 85 f6 movslq %r11d,%r11 lea (%rdi,%r11,4),%rax lea (%r10,%r11,4),%r11 movaps %xmm0,%xmm1 xor %r15d,%r15d xor %r14d,%r14d xor %r13d,%r13d test %esi,%esi This is the output of „objdump –d a.out“ for some binary a.out. Try yourself … 1. FETCH: get a binary number from memory 2. DECODE: translated a binary number into a meaningful instruction

April Pipelines – Execution and Memory Access So we can fetch the next number and translate it into a meaningful instruction. What now? If the instruction is arithmetic, just execute it, e.g. mul %r14d,%r15d (multiply register 14 with register 15) If it is a memory reference, load the data, e.g. mov,%r15d Some memory references involve computation (arithmetic) lea (%r10,%r11,4),%r11 3. EXECUTE: Execute the instruction 4. MEMORY ACCESS: Load and store data into memory, maybe compute the address in step 3.

April Pipelines – Write back This is the final step of our primitve pipeline Once the the result is computed or loaded, write it into the register specified, e.g. mul %r14d,%r15d Put the result here The need for this step is not immediately clear, but will become important later on … 5. WRITE BACK: Save the result as specified

April Our CPU construction – functional parts 1. Fetch a new instruction 2. Decode Instruction 3. Exectution 4. Memory Access 5. Write Register Data Let‘s assume nothing for the moment – The CPU is just a black box with data coming in and data going out

April … a CPU with 5 pipeline stages Inst Data Instr. Fetch Instr. Decode Execute Memory Access Write Back Data

April Pipelining 1 – No Pipelining Execution time = N instructions * T p Fetch Instruction Decode Instruction Execute Write Back Memory Access FDEMW T p

April Processor Technology Pipelining 1 – No Pipelining The concept of a non-pipelinging CPU is very inefficient! While, e.g. some comparison is performed in the ALU, the other functional units are running idle. With very little effort we can use all functional units at the same time and increase the performance tremendously Pipeling let‘s different commands use different functional units at the same time. There are problems connected to pipelined operation. We will address them later.

April Pipelining 2 - Pipelining Fetch Instruction Decode Instruction Execute Write Back Memory Access FDEMW T p Execution time = N instructions * T s T s

April Our Core2 Construction Inst Data Instr. Fetch Instr. Decode ALU Memory Access Write Registers Data

April Pipelining 3 – Pipeline Stalls 1 FDEMW 32 R Example code:... c=a/b d=c/b e=c/d f=e/d g=f/e... Pipeline Stalls can have two reasons – competition for resources and data dependencies

April Pipelining 4 – Branches FDEM W 3 JZ5JZ5 CMPCMP 45 Branches are a consicerable obstacle for performance! F Compare two values and write result in F Jump if Flag F is 0

April Processor Technology Avoiding Pipeline Stalls – Branch Prediction Pipeline stalls due to branches can be avoided by introducing a functional unit the „guesses“ the next branch. 00 No jump predicte d No jump predicte d Jump predicte d 11 Jump predicte d jump no jump jump no jump jump no jump

April Pipelining 4 – Branch Prediction FDEM W 3 JZ5JZ5 CMPCMP 45 Branch prediction units can predict with a probability of >>90%. At a wrongly predicted branch the pipeline needs to be flushed! F Compare two values and write result in F Jump if Flag F is 0 Branch Target Buffer

April Our CPU Construction Inst Data Instr. Fetch DecodeALU Memory Access Write Registers Data Branch Target Buffer

April Superscalar Processors 1 The superscalar concept is an extension of the pipeline concept Pipelining alows for „hiding“ the latency of an instruction Pipelining idealy achieves 1 instruction per clock cycle Higher ratest can only be achieved if more than one instruction can be executed at the same time 1.Superscalar architecture - Xeon 2.Other idea: Very long instruction word (VLIW) – EPIC/Itanium

April Superscalar Processors 2 Fetch Instruction + Decode Dispatch ALU Write Register Memory Access FDEMW Execution time = N instructions * T s / N Pipelines Problems of the Pipeline remain!

April Processor Technology Superscalar Processors 3 The superscalar concept dramatically improves performance (2x at best) The different pipelines don‘t need to have the same functionalities The superscalar concept can be extended easily

April Our CPU Construction Inst Data Instr. Fetch + Decode Dispatch ALU Memory Access Write Registers Data Branch Target Buffer ALU Read Registers Write Registers

April Processor Technology Out-Of-Order Execution Speculative computing provides an important way of reducing pipeline stalls in case of branches. Still data dependencies cause pipeline stalls There is no good reason why the CPU shouldn‘t execute instruction which don‘t have dependencies or for which the dependencies are already computed. Out-Of-Order a way to minimize the pipeline stalls by reordering the instructions at entry to the pipeline and restore the original order when exeting the pipeline

April Processor Technology Out-Of-Order Execution 1 FRAM W 3 2 R 45 Inst. Queue Reorder Buffer

April Our Construction Inst Data Instr. Fetch + Decode Dispatch ALU Memory Access Data Branch Target Buffer ALU Read Registers Reorder Buffer Retire - Write Registers

April Processor Technology Register Allocation IA32 provides a very limited number of registers for local storage This is only due to definition in the instruction set architecture (ISA) Internally there are more registers available The CPU can internally re-assign the registers in order make best use of the given resources. This re-allocation is tracked in the Register Allocation Table

April Our CPU Construction Inst Data Instr. Fetch Dispatch ALU Memory Access Data Branch Target Buffer ALU Read Registers Reorder Buffer Retire - Write Registers Register Allocation Now let‘s compare to the original...

April Block Diagram Branch Target Buffer Microcode Sequencer Register Allocation Table (RAT) 32 KB Instruction Cache Next IP Instruction Decode (4 issue) Fetch / Decode Retire (Write back) Re-Order Buffer (ROB) – 128 entry IA Register Set To L2 Cache Port Bus Unit Reservation Stations (RS) 32 entry Scheduler / Dispatch Ports 32 KB Data Cache Execute Port FP Add SIMD Integer Arithmetic Memory Order Buffer (MOB) Load Store Addr FP Div/Mul Integer Shift/Rotate SIMD Integer Arithmetic Port Store Data Not very different, right? Memory Dispatch

April Summary Nehalem can be thought of very much in the same as standard text books present a CPU 1.Instruction Fetch and Decode (Frontend) 2.Dispatch 3.Execute 4.Memory Access 5.Retire (Write Back) If you want to dive deeper into the subject: Hennessy and Patterson: Computer Architecture – A quantitave approach

April