1. 1 Chapter 5 Cache Memory Chapter Five : Cache Memory Memory Processor Input/Output

2. 2 Characteristics of memory systems  Location  Capacity  Unit of transfer  Access method  Performance  Physical type  Physical characteristics  Organization Chapter Five : Cache Memory

3. 3 Location  Internal memory is often equated with main memory.  The processor requires its own local memory in the form of registers e.g. PC,IR,ID…..  The control unit portion of the processor may also require its own internal memory.  Cache is another form of internal memory  External memory consists of peripheral storage devices e.g. disk, tape, that are accessible to the processor via I/O controller. Chapter Five : Cache Memory

4. 4 Capacity  Word size —The natural unit of organization —Typically equal to the number of bits used to represent integer and instruction length. —Common word lengths are 8, 16 and 32 bits. —External memory capacity is typically expressed in terms of bytes (1GB, 20MB) Chapter Three : Cache Memory

5. 5 Unit of Transfer  Internal memory —Number of bits read out of or written into memory at a time. —The unit of transfer is equal to the number of data lines of the memory module. —Equal to the word length but is often larger, such as 64,128 or 256 bits.  External memory —Usually a block which is much larger than a word and these are referred as blocks.  Addressable unit —Smallest location which can be uniquely addressed —Word internally Chapter Three : Cache Memory

6. 6 Access Methods (1)  Sequential —Start at the beginning and read through in order —Access time depends on location of data and previous location —e.g. tape  Direct —Individual blocks have unique address —Access is by jumping to vicinity plus sequential search (pointer) —Access time depends on location and previous location —e.g. disk Chapter Three : Cache Memory

7. 7 Access Methods (2)  Random —Individual addresses identify locations exactly —Access time is independent of location or previous access —e.g. RAM  Associative —Data is located by a comparison with contents of a portion of the store —Access time is independent of location or previous access —e.g. cache Chapter Three : Cache Memory

8. 8 Performance  Access time (latency) – for RAM, this is the time it takes to perform a read or write operation, that is, the time from the instant that an address is presented to the memory to the instant that data have been stored or made available for use.  Memory cycle time – consists of the access time plus any additional time required before a second access can commence. This additional time may be required for transients to die out on signal lines or to regenerate data if they are read destructively.  Transfer rate – this is the rate that data can be transferred into or out of a memory unit. Note: Two most important characteristics of memory : performance and capacity Chapter Three : Cache Memory

9. 9 Physical Types of memory  Semiconductor —RAM  Magnetic —Disk & Tape  Optical —CD & DVD  Others —Bubble —Hologram Chapter Three : Cache Memory

10. 10 Physical Characteristics  Volatile – information decays naturally or is lost when electrical power is switched off.  Non volatile – information once recorded remains without deterioration until deliberately changed and no electrical electric power is needed to retain information. e.g magnetic- surface memory.  Non erasable – cannot be altered, except by destroying the storage unit. E.g. ROM.  Power consumption Chapter Three : Cache Memory

11. 11 Cost, capacity and access time There is a trade-off among the three key characteristics of memory (cost, capacity and access time)  Faster access time, greater cost per bit  Greater capacity, smaller cost per bit  Greater capacity, slower access time Chapter Three : Cache Memory

12. 12 Memory Hierarchy  Registers —In CPU  Internal or Main memory —May include one or more levels of cache —“RAM”  External memory —Backing store Chapter Three : Cache Memory

13. 13 Memory Hierarchy - Diagram Chapter Three : Cache Memory

14. 14 Memory Hierarchy - Diagram (Cont’d) Based on the figure, as one goes down the hierarchy the following occur — Decreasing cost per bit — Increasing capacity — Increasing Access time — Decreasing the frequency of access of the memory by the processor Chapter Three : Cache Memory

15. 15 Memory Organization Smaller, more expensive, faster memories are supplemented by larger, cheaper, slower memories. The key to the success of this organization is decreasing frequency of access. This concept is explain detail in discussing on  cache memory ( next slide )  virtual memory Chapter Three : Cache Memory

16. 16 The Bottom Line  How much? —Capacity  How fast? —Time is money  How expensive? Chapter Three : Cache Memory

17. 17 Hierarchy List  Registers  L1 Cache  L2 Cache  Main memory  Disk cache  Disk  Optical  Tape Chapter Three : Cache Memory

18. 18 Locality of Reference  During the course of the execution of a program, memory references tend to cluster  e.g. loops Chapter Three : Cache Memory

19. 19 Cache-Introduction  Small amount of fast memory  Sits between normal main memory and CPU  May be located on CPU chip or module Chapter Three : Cache Memory Small capacity, fast Large capacity, slow

20. 20 Cache-Introduction  Is an intermediate buffer between CPU and main memory.  Objective :- to reduce the CPU waiting time during the main memory accesses.  Without cache memory, every main memory access results in delay in instruction processing.  Due to main memory access time is higher than processor clock period.  High speed CPU’s time wasted during memory access (instruction fetch, operand fetch or result storing)  To minimize the waiting time for the CPU, a small but fast memory is introduced as a cache buffer between main memory and CPU. Chapter Three : Cache Memory

21. 21  A portion of program and data is brought into the cache memory in advance.  The cache controller keeps track of locations of the main memory which are copied In cache memory  When CPU needs for instruction or operand, it receives from the cache memory (if available) instead of accessing the main memory.  Thus the memory access is fast because the slow main memory appears as a fast memory.  The cache memory capacity is very small compared to main memory but the speed is several times better than main memory.  Transfer between the CPU and cache memory usually one word at a time.  The cache memory usually can be physically with processor IC as internal cache, also known as on-chip cache. Chapter Three : Cache Memory

22. 22 Cache/Main Memory Structure Chapter Three : Cache Memory Cache consists of C-lines. Each line contains K words and a tag of a few bits. The number of words in the line referred as line size. Tag- a portion of main memory address. Each line includes a tag identifies which particular block is currently being stored.

23. Remember Unit Value  KB=2 10 Bytes = 1024 Bytes  MB=2 20 Bytes = 1024 KB  GB=2 30 Bytes = 1024 MB  TB=2 40 Bytes = 1024 GB 23

24. 24 Example Cache Memory  Cache of 64KBytes  Cache block of 4 bytes each. —64K/4=64(1024)bytes/4bytes=16 K= 2 14 lines —i.e. cache is 16K =(2 14 ) lines of 4 bytes each Main Memory  16MBytes main memory —(2 4 )(2 20 )bytes= (2 24 )=16MBytes —24 bit address Chapter Three : Cache Memory Note : 1 Kilobytes = 1,024 bytes or (2 10 ),

25. 25 Cache operation – overview  CPU requests contents of memory location  Check cache for this data  If present, get from cache (fast)  If not present, read required block from main memory to cache  Then deliver from cache to CPU  Cache includes tags to identify which block of main memory is in each cache slot Chapter Three : Cache Memory

26. 26 Cache Read Operation - Flowchart Chapter Three : Cache Memory

27. 27 Cache Design  Size  Mapping Function  Replacement Algorithms  Write Policy  Line Size  Number of Caches Chapter Three : Cache Memory

28. 28 Size does matter  Cost —More cache is expensive  Speed —More cache is faster (up to a point) —Checking cache for data takes time Chapter Three : Cache Memory

29. 29 Typical Cache Organization Chapter Three : Cache Memory

30. 30 Comparison of Cache Sizes a Two values separated by a slash refer to instruction and data caches b Both caches are instruction only; no data caches ProcessorType Year of Introduction L1 cache a L2 cacheL3 cache IBM 360/85Mainframe196816 to 32 KB—— PDP-11/70Minicomputer19751 KB—— VAX 11/780Minicomputer197816 KB—— IBM 3033Mainframe197864 KB—— IBM 3090Mainframe1985128 to 256 KB—— Intel 80486PC19898 KB—— PentiumPC19938 KB/8 KB256 to 512 KB— PowerPC 601PC199332 KB—— PowerPC 620PC199632 KB/32 KB—— PowerPC G4PC/server199932 KB/32 KB256 KB to 1 MB2 MB IBM S/390 G4Mainframe199732 KB256 KB2 MB IBM S/390 G6Mainframe1999256 KB8 MB— Pentium 4PC/server20008 KB/8 KB256 KB— IBM SP High-end server/ supercomputer 200064 KB/32 KB8 MB— CRAY MTA b Supercomputer20008 KB2 MB— ItaniumPC/server200116 KB/16 KB96 KB4 MB SGI Origin 2001High-end server200132 KB/32 KB4 MB— Itanium 2PC/server200232 KB256 KB6 MB IBM POWER5High-end server200364 KB1.9 MB36 MB CRAY XD-1Supercomputer200464 KB/64 KB1MB—

31. 31 Mapping Function  Fewer cache lines than the main memory block.  Therefore, an algorithm is needed for mapping main memory blocks into cache lines.  Needs for determining which memory block currently occupies a cache line--  Mapping function  The choice of the mapping function, determines the how the cache is organized.  Three techniques :- —Direct —Associative —Set associative Chapter Three : Cache Memory

32. Direct mapping cache example  In a direct mapping cache with a capacity of 16 KB and a line length of 32 bytes, determine the following; i) How many bits are used to determine the byte that a memory operation references within the cache line? ii) How many bits are used to select the line in the cache that may contain the data. 32

33. 33 Direct Mapping  Each block of main memory maps to only one cache line —i.e. if a block is in cache, it must be in one specific place  Address is in two parts  Least Significant w bits identify unique word  Most Significant s bits specify one memory block  The MSBs are split into a cache line field r and a tag of s-r (most significant) Chapter Three : Cache Memory

34. 34 Direct Mapping Address Structure : Example TagLine or SlotWord s-r rw  EXAMPLE : TAG=8, LINE = 14, WORD=2  24 bit address  2 bit word identifier (4 byte block)  22 bit block identifier —8 bit tag (=22-14) —14 bit slot or line  No two blocks in the same line have the same Tag field  Check contents of cache by finding line and checking Tag

35. 35 Direct Mapping Cache Line Table Cache lineMain Memory blocks held 00, m, 2m, 3m…2s-m 11,m+1, 2m+1…2s-m+1... m-1m-1, 2m-1,3m-1…2s-1

36. 36 Direct Mapping Cache Organization

37. 37 Direct Mapping Example

38. 38 Direct Mapping Summary  Address length = (s + w) bits  Number of addressable units = 2s+w words or bytes  Block size = line size = 2w words or bytes  Number of blocks in main memory = 2s+ w/2w = 2s  Number of lines in cache = m = 2r  Size of tag = (s – r) bits

39. 39 Direct Mapping pros & cons  Simple  Inexpensive  Fixed location for given block —If a program accesses 2 blocks that map to the same line repeatedly, cache misses are very high

40. 40 Associative Mapping  A main memory block can load into any line of cache  Memory address is interpreted as tag and word  Tag uniquely identifies block of memory  Every line’s tag is examined for a match  Cache searching gets expensive

41. 41 Fully Associative Cache Organization

42. 42 Associative Mapping Example

43. 43 Tag 22 bit Word 2 bit Associative Mapping Address Structure  22 bit tag stored with each 32 bit block of data  Compare tag field with tag entry in cache to check for hit  Least significant 2 bits of address identify which 16 bit word is required from 32 bit data block  e.g. —AddressTagDataCache line —FFFFFCFFFFFC246824683FFF

44. 44 Associative Mapping Summary  Address length = (s + w) bits  Number of addressable units = 2s+w words or bytes  Block size = line size = 2w words or bytes  Number of blocks in main memory = 2s+ w/2w = 2s  Number of lines in cache = undetermined  Size of tag = s bits

45. 45 Set Associative Mapping  Cache is divided into a number of sets  Each set contains a number of lines  A given block maps to any line in a given set —e.g. Block B can be in any line of set i  e.g. 2 lines per set —2 way associative mapping —A given block can be in one of 2 lines in only one set

46. 46 Set Associative Mapping Example  13 bit set number  Block number in main memory is modulo 2 13  000000, 00A000, 00B000, 00C000 … map to same set

47. 47 Two Way Set Associative Cache Organization

48. 48 Set Associative Mapping Address Structure  Use set field to determine cache set to look in  Compare tag field to see if we have a hit Tag 9 bit Set 13 bit Word 2 bit

49. 49 Two Way Set Associative Mapping Example

50. 50 Set Associative Mapping Summary  Address length = (s + w) bits  Number of addressable units = 2s+w words or bytes  Block size = line size = 2w words or bytes  Number of blocks in main memory = 2d  Number of lines in set = k  Number of sets = v = 2d  Number of lines in cache = kv = k * 2d  Size of tag = (s – d) bits

51. 51 Comparison Cache TypeHit RatioSearch Speed Direct MappedGoodBest Fully AssociativeBestModerate N-Way AssociativeVery Good, Better as N increases Good, Worse as N increases

52. 52 ASSIGNMENT # 3  Direct Mapping Technique. —A 32 bit computer has 32 bit memory address. It has 8KB of cache memory. Each line size is 16 bytes. Show the memory address format.  Associative Mapping Technique. —A 32 bit computer has 32 bit memory address. Each line size is 16 bytes. Show the memory address.  Set Associative Mapping Technique. —A 32 bit computer has 32 bit memory address. It has 8KB of cache memory. The computer follows the four- way set associative mapping. Each line set is 16 bytes. Show the memory address format. Chapter Three : Cache Memory

53. 53 Replacement Algorithms (1) Direct mapping  No choice  Each block only maps to one line  Replace that line

54. 54 Replacement Algorithms (2) Associative & Set Associative  Hardware implemented algorithm (speed)  Least Recently used (LRU)  e.g. in 2 way set associative —Which of the 2 block is lru?  First in first out (FIFO) —replace block that has been in cache longest  Least frequently used —replace block which has had fewest hits  Random

55. 55 Write Policy  Must not overwrite a cache block unless main memory is up to date  Multiple CPUs may have individual caches  I/O may address main memory directly

56. 56 Write through  All writes go to main memory as well as cache  Multiple CPUs can monitor main memory traffic to keep local (to CPU) cache up to date  Lots of traffic  Slows down writes  Remember bogus write through caches!

57. 57 Write back  Updates initially made in cache only  Update bit for cache slot is set when update occurs  If block is to be replaced, write to main memory only if update bit is set  Other caches get out of sync  I/O must access main memory through cache  N.B. 15% of memory references are writes

58. 58 Pentium 4 Cache  80386 – no on chip cache  80486 – 8k using 16 byte lines and four way set associative organization  Pentium (all versions) – two on chip L1 caches —Data & instructions  Pentium III – L3 cache added off chip  Pentium 4 —L1 caches 8k bytes 64 byte lines four way set associative —L2 cache Feeding both L1 caches 256k 128 byte lines 8 way set associative —L3 cache on chip

59. 59 Intel Cache Evolution ProblemSolution Processor on which feature first appears External memory slower than the system bus. Add external cache using faster memory technology. 386 Increased processor speed results in external bus becoming a bottleneck for cache access. Move external cache on-chip, operating at the same speed as the processor. 486 Internal cache is rather small, due to limited space on chip Add external L2 cache using faster technology than main memory 486 Contention occurs when both the Instruction Prefetcher and the Execution Unit simultaneously require access to the cache. In that case, the Prefetcher is stalled while the Execution Unit’s data access takes place. Create separate data and instruction caches. Pentium Increased processor speed results in external bus becoming a bottleneck for L2 cache access. Create separate back-side bus that runs at higher speed than the main (front-side) external bus. The BSB is dedicated to the L2 cache. Pentium Pro Move L2 cache on to the processor chip. Pentium II Some applications deal with massive databases and must have rapid access to large amounts of data. The on-chip caches are too small. Add external L3 cache.Pentium III Move L3 cache on-chip.Pentium 4

60. 60 Pentium 4 Block Diagram

61. 61 Pentium 4 Core Processor  Fetch/Decode Unit —Fetches instructions from L2 cache —Decode into micro-ops —Store micro-ops in L1 cache  Out of order execution logic —Schedules micro-ops —Based on data dependence and resources —May speculatively execute  Execution units —Execute micro-ops —Data from L1 cache —Results in registers  Memory subsystem —L2 cache and systems bus

62. 62 Pentium 4 Design Reasoning  Decodes instructions into RISC like micro-ops before L1 cache  Micro-ops fixed length —Superscalar pipelining and scheduling  Pentium instructions long & complex  Performance improved by separating decoding from scheduling & pipelining —(More later – ch14)  Data cache is write back —Can be configured to write through  L1 cache controlled by 2 bits in register —CD = cache disable —NW = not write through —2 instructions to invalidate (flush) cache and write back then invalidate  L2 and L3 8-way set-associative —Line size 128 bytes

63. 63 PowerPC Cache Organization  601 – single 32kb 8 way set associative  603 – 16kb (2 x 8kb) two way set associative  604 – 32kb  620 – 64kb  G3 & G4 —64kb L1 cache 8 way set associative —256k, 512k or 1M L2 cache two way set associative  G5 —32kB instruction cache —64kB data cache

64. 64 PowerPC G5 Block Diagram

65. 65 Layers of Cache LevelDevices Cached Level1Level2 cache, system RAM, Hard Disk, DC-ROM Level2system RAM, Hard Disk, DC-ROM System RAMHard Disk, DC-ROM Hard Disk/CD-ROM--

66. 66 More about Cache  Disk cache, peripheral cache (e.g.: CDs, very slow initial access time), Internet cache (software).  A 512kb level 2 cache, caching 64MB of system memory, can supply 90-95% of info. <=1% in size. This is due to locality of reference (large programs with several MBs of instructions, only small portions are loaded at a time (e.g.: Opening a word document)).  SRAM  7-20 ns, DRAM  50-70 ns  Level2  256-512KB, level1  8-64KB

67. 67 Locality of reference  Special Locality: Reflects the tendency of the processor to access data locations sequentially.  Temporal Locality: Reflects the tendency of the processor to access the recently used memory locations.