1. Anshul Kumar, CSE IITD CSL718 : Main Memory CPU-Cache-Main Memory Performance 9th Mar, 2006

2. Anshul Kumar, CSE IITD slide 2 A Simple Model t av = t c + p m. t c.miss where t av = average memory access time as seen by CPU t c = cache access time p m = miss probability (consider only read misses, if write penalties are hidden by buffers) t c.miss = cache miss penalty CPUCache Memory

3. Anshul Kumar, CSE IITD slide 3 Cache miss penalty Depends on Various cache policies –Read policy –Load policy –Write policy –Write buffers etc. Main memory organization –Interleaving –Page mode

4. Anshul Kumar, CSE IITD slide 4 Read Policies Cache Memory T eff =(1-p m ).1 + p m. (T+2) Sequential Simple: Cache Memory T eff =(1-p m ).1 + p m. (T+1) Concurrent Simple: Cache Memory T eff =(1-p m ).1 + p m. (T+1) Sequential Forward: Cache Memory T eff =(1-p m ).1 + p m. (T) Concurrent Forward: 111 T 111 T 11 T 11 T

5. Anshul Kumar, CSE IITD slide 5 Load policies 4 AU Block Cache miss on AU 1 Block Load Load Forward Fetch Bypass (wrap around load) 0 1 2 3

6. Anshul Kumar, CSE IITD slide 6 Analyzing Write Policies:CPU time Hit:WB, Miss: WB1T b + i11 Hit:WB, Miss: WTWA1T b + i11 Hit:WB, Miss: WTNWA1T b + i11 Hit:WT, Miss: WB1T b + i11 Hit:WT, Miss: WTWA1T b + i11 Hit:WT, Miss: WTNWA1T b + i11 PolicyReadReadWriteWrite hitmisshitmiss i depends on read policy

7. Anshul Kumar, CSE IITD slide 7 Analyzing Write Policies:Bus time Hit:WB, Miss: WB0T b (2-Pc)0 T b (2-Pc) Hit:WB, Miss: WTWA0T b (2-Pc) 0 T b (2-Pc)+T w Hit:WB, Miss: WTNWA0T b (2-Pc) 0 T w Hit:WT, Miss: WB0T b (2-Pc) T w T b (2-Pc) Hit:WT, Miss: WTWA0T b T w T b +T w Hit:WT, Miss: WTNWA0T b T w T w PolicyReadReadWriteWrite hitmisshitmiss

8. Anshul Kumar, CSE IITD slide 8 Interleaving with Fast Page Mode

9. Anshul Kumar, CSE IITD slide 9 A Refined Model t av = t c + p m. (t c.miss + t interference + t w-interference + t IO-interference ) where t interference = interference among line transfers t w-interference = interference between word writes and line transfers t IO-interference = interference between I/O and line transfers

10. Anshul Kumar, CSE IITD slide 10 Interference among line transfers What happens when another miss occurs in t busy = t m.miss - t c.miss interval? t interference = additional delay due to this = expected number of misses during t busy * delay per miss = ( * t busy * p m ) * (t busy / 2) where = memory request rate of processor tctc t c.miss t m.miss CPU blocked CPU executing Memory busy

11. Anshul Kumar, CSE IITD slide 11 Interference I/Os and writes delay= prob that memory is busy when request arrives * average waiting period what happens when memory is found to be busy serving one request and some other requests are waiting? Memory busy request arrivals served waiting served

12. Anshul Kumar, CSE IITD slide 12 I/O Interference t IO-interference = delay due to I/O contention = probability that memory is occupied with I/O * average time taken to complete ongoing I/O = (  ) * (t service +t IO-wait )/2 t service = time to service (block read/write time) t IO-wait = waiting time = 0, if CPU has a higher priority  0, otherwise  estimate using queuing model

13. Anshul Kumar, CSE IITD slide 13 Write Interference Delay t w-interference = probability that a write through is occupying the memory when a read miss occurs * average time taken to complete ongoing write

14. Anshul Kumar, CSE IITD slide 14 Memory performance using queuing model Arrival of requests (from processor/cache) Servicing of requests (by memory) Requests queued for service Statistical behaviour of arrivals ? Statistical behaviour of service? Model Nomenclature: arrival / service / number M / G / 1G : General M / M / 1M : Poisson/Exponential M / D / 1D : Constant M B / D / 1M B : Binomial

15. Anshul Kumar, CSE IITD slide 15 Modeling memory requests prob of a request in one cycle = p prob of no request in one cycle = 1 – p prob of no request in T/  cycles = (1 – p) T/  prob of at least one req in T/  cycles = 1 – (1 – p) T/  prob of k requests in n (=T/  ) cycles = n C k p k (1 – p) n-k (Binomial distribution) expected no. of requests in n cycles = n p T : interval (memory cycle time)  : processor cycle

16. Anshul Kumar, CSE IITD slide 16 Poisson Approximation If processor cycles are small (i.e.,   0, p  0, n  , n p  T), Binomial distribution  Poisson distribution, request rate = prob of k requests in interval T = expected no. of requests in interval T = T Interval between two consecutive requests has an exponential distribution, prob (inter arrival interval > t) = 1 – e - t

17. Anshul Kumar, CSE IITD slide 17 Modeling Service Each request is served in constant time e.g. cache write through requests, cache block transfer requests or Service time has an exponential distribution e.g. I/O requests with varying block sizes where small blocks are more common than large blocks

18. Anshul Kumar, CSE IITD slide 18 M / G / 1 Model Average waiting time = T w = Average queue length = Q = where  = occupancy of server = /   = average service rate c =    = variance of service time

19. Anshul Kumar, CSE IITD slide 19 Special cases: M/M/1, M/D/1 M/M/1  c = 1 Average waiting time = T w = Average queue length = Q = M/D/1  c = 0 Average waiting time = T w = Average queue length = Q =

20. Anshul Kumar, CSE IITD slide 20 M/D/1 with low server occupancy Average waiting time = T w = Average queue length = Q = when  is small, T w = = Compare this with

21. Anshul Kumar, CSE IITD slide 21 Designing buffer to hold the queue How to design a buffer so that buffer overflow or stalling due to buffer full is within certain limit? For M/M/1 model, prob(queue size  buffer size BF) =  BF+1 Choose BF so that this probability is below a desired value.

22. Anshul Kumar, CSE IITD slide 22 Open and Closed Queues Arrival of requests (from processor/cache) Servicing of requests (by memory) Requests queued for service Processor is not blocked by queuing delays and request rate remains unaffected – Open queue Processor is blocked due to queuing delays and request rate drops – Closed queue

23. Anshul Kumar, CSE IITD slide 23 Open and Closed Queues Arrival of requests (from processor/cache) Servicing of requests (by memory) Requests queued for service Time Tw 1/  Number (open) Q = T w  = /  Number (closed) Q a  a occupancy (open q) =  = occupancy (closed q) + waiting (closed q)  a + Q a

24. Anshul Kumar, CSE IITD slide 24 M/D/1 Closed Queue Reduced request rate = a Reduced occupancy =  a = a /  Requests being served =  a Requests waiting =

25. Anshul Kumar, CSE IITD slide 25 Deriving queue length, wait time Lett i = time when request i is being served r i = no. of arrivals during t i n i = queue length at the end of t i including item in service Assume occupancy of server =  = /  < 1  process reaches a steady state Expected valueE(t i ) = E(t ) = T = 1/  E(r i ) = E(r ) = E(t ) = /  =  E(n i ) = E(n ) = N

26. Anshul Kumar, CSE IITD slide 26 Relating n i+1 to n i n i+1 = n i + arrivals – departures two cases need to be considered: i) n i  0 ii) n i = 0 C i+1 C i+2 C i+3  CiCi nini

27. Anshul Kumar, CSE IITD slide 27 When n i  0 C i+1 arrived before C i left n i+1 = n i + r i+1 - 1 C i servedC i+1 served C i leaves C i+1 leaves time titi t i+1

28. Anshul Kumar, CSE IITD slide 28 When n i = 0 C i+1 arrived after C i left n i+1 = n i + 1 + r i+1 – 1 = n i + r i+1 C i servedC i+1 served C i leaves C i+1 leaves time titi t i+1 C i+1 arrives

29. Anshul Kumar, CSE IITD slide 29 Combining the two cases n i+1 = n i + r i+1 – 1 +  i where  i = 0, when n i  0 and  i = 1, when n i = 0 note that n i  i = 0 and  i 2 =  i E(n i+1 ) = E( n i ) + E( r i+1 ) – 1 + E(  i ) in steady state, E(n) = E( n ) + E( r ) – 1 + E(  ) that is, E(  ) = 1 - E( r ) = 1 -  prob ( n  0) = 

30. Anshul Kumar, CSE IITD slide 30 Combining the two cases n i+1 = n i + r i+1 – 1 +  i n i+1 2 = n i 2 + (r i+1 – 1) 2 +  i 2 + 2 n i (r i+1 – 1) + 2 (r i+1 – 1)  i + 2 n i  i n i+1 2 = n i 2 + (r i+1 – 1) 2 +  i + 2 n i (r i+1 – 1) + 2 (r i+1 – 1)  i E(n i+1 2 ) = E( n i 2 ) + E(r i+1 – 1) 2 + E(  i ) + 2 E[ n i (r i+1 – 1) ] + 2 E[(r i+1 – 1)  i ] 0 = E[(r – 1) 2 ] + E(  ) + 2 E[ n (r – 1) ] + 2 E[(r – 1)  ] 0 = E(r 2 )-2  +1+ (1-  ) + 2 E(n) (  – 1) + 2 (  – 1)(1-  ) 2 E(n) (1-  ) = E(r 2 ) -2  2 + 

31. Anshul Kumar, CSE IITD slide 31 continuedcontinued 2 E(n) (1-  ) = E(r 2 ) -2  2 +  This is valid for G/G/1

32. Anshul Kumar, CSE IITD slide 32 Consider Poisson arrival P(r i ) = mean E(r i ) = t i variance  ri 2 = t i  ri 2 = E(r i 2 ) - |E(r i )| 2 E(r i 2 ) =  ri 2 + |E(r i )| 2 Take expectation over i E(r 2 ) = E(t) + 2 E(t 2 )

33. Anshul Kumar, CSE IITD slide 33 continuedcontinued mean E(t) = 1/  variance  t 2 E(t 2 ) =  t 2 + [E(t) ] 2 =  t 2 + 1/  2 Recall E(r 2 ) = E(t) + 2 E(t 2 ) Therefore, E(r 2 ) = /  + 2 (  t 2 + 1/  2 ) =  + 2  t 2 +  2 where c 2 =  2  t 2

34. Anshul Kumar, CSE IITD slide 34 Direct Derivation for M/M/1 P(n; t) = prob that there are n req in the system at time t (in queue + in service) P(n; t+  t) = P(n; t)(1 -  t -   t) + P(n-1; t)  t + P(n+1; t)   t P(0; t+  t) = P(0; t)(1 -  t) + P(1; t)   t Prob of more than one event in  t is neglected (  t 2 term)

35. Anshul Kumar, CSE IITD slide 35 Direct Derivation for M/M/1 dP(n; t)/dt = P(n; t)(- -  ) + P(n-1; t) + P(n+1; t)  dP(0; t)/dt = P(0; t)(- ) + P(1; t)  In steady state, We can drop ;t Derivatives tend to 0  0 = P(n)(- -  ) + P(n-1) + P(n+1)  0 = P(0)(- ) + P(1)   P(n) -  P(n+1) = P(n-1) -  P(n) P(0) -  P(1) = 0  P(n-1) -  P(n) = 0  P(n) =  P(n-1)

36. Anshul Kumar, CSE IITD slide 36 Direct Derivation for M/M/1 P(n) =  P(n-1)  P(n) =  n P(0)

37. Anshul Kumar, CSE IITD slide 37 Direct Derivation for M/M/1