1. Review Session Jehan-François Pâris

2. Agenda Statistical Analysis of Outputs Operational Analysis Case Studies Linear Regression

3. How to use this presentation Most problems have  One slide stating the problem  One slide explaining how to solve the problem  One slide allowing you to check your answer You will learn more by trying first to do the problems on your own than by reading their solutions Do not forget either to review the problems in the original notes

4. Statistical Analysis of Outputs

5. The big picture The problems  Constructing confidence intervals  Handling auto correlated data The tools  Central-Limit Theorem  Wilson’s formula  Batch means (and regeneration)  RNG tricks

6. Confidence Intervals Distinguish between  CIs for means CSIM does it for you  CIs for proportions We are on our own Major issue is independence of data points CSIM uses batch means

7. Central Limit Theorem If the n mutually independent random variables x 1, x 2, …, x n have the same distribution, and if their mean  and their variance  2 exist then …

8. Central Limit Theorem The random variable is distributed according to the standard normal distribution (zero mean and unit variance).

9. CI for means (I) For large values of n, the (1-  )% confidence interval for  is given by with

10. CI for means (II) F(z) is taken from a table of the normal distribution  F(0.025) = 1.96 For smaller values of n, we have to use Student’s t random variable  Wider CIs We replace  by the sample standard deviation s

11. Example We have  100 observations for the waiting time  xbar = 4.25 minutes  s 2 = 25

12. Example We have  100 observations for the waiting time  xbar = 4.25 minutes  s 2 = 25 Answer is  4.25 ± 1.96 sqrt(25/100) = 4.25 ± 0.98

13. CI for proportions A proportion represents the probability P(X  ) for some fixed threshold  97% of our customers have to wait less than one minute Distributed according to a binomial law  Use Wilson ’ s formula

14. Wilson’s formula When n > 29, we can use the Wilson’s interval where z  /2 = 1.96 for a 95% C.I.

15. Example We have want to estimate the proportion of packets that wait more than four slots  400 observations  40 packets waited more than four slots

16. Answer Divisor:  1 + 1.96 2 /400  1.01 (instead of 1.0096) Central term  0.1 + 1.96 2 /(2×400)  0.105 (instead of 1.048) Half width  sqrt( (0.1×0.9)/400 + 1.96 2 /(2×400 2 ) )  sqrt (0.09/400 + (4/800)/400)  1/20 sqrt (0.09 +0.0025)  0.3/20 = 0.015 Result is  (0.105 ± 0.015)/ 1.01 = 0.104 ± 0.015

17. Batch means (I) Simulation data are often autocorrelated  Packet delays in ALOHA  Waiting times in queues  … Batch means reduce (but do not completely eliminate) that effect

18. Batch means (II) Group measurements into fixed-size batches of consecutive data Compute mean of each batch If batches are large enough, these means will be independent  Can use standard-limit theorem, … In case of doubt, compute autocorrelation function for successive batch means

19. Regeneration (I) The idea  Partition simulation data into intervals such that Data measured inside the same interval might be correlated Data measured in different intervals are independent

20. Regeneration (II) How?  System goes to a regeneration point each time Its queues become empty All the disk drives are operational …  Criterion is system specific

21. Streams When you want to evaluate two different configurations of a system, it is often good idea to use separate random number streams for arrivals and service times  Arrival times remain unchanged when we change other parameters of the system

22. Operational Analysis

23. Single server (I) We can measure  T the length of the observation period  A the number of arrivals during the observation period  B the total amount of busy times during the observation period  C the number of completions during the observation period

24. Single server (II) We can compute  = A/Tthe arrival rate  X = C/Tthe output rate  U = B/Tthe utilization  S = B/Cthe mean service time There are two ways to compute U  U = B/T = (C/T )(B/C) = XS In general A  C and  X

25. Little’s law If W is the total time spent by all tasks inside the system over the observation period, then  N = W/T  R = W/C Since W/T = (C/T)(W/C) = XR, N = XR This is important

26. A problem An ice-cream parlor  Observed during 6 hours  Visited by 120 customers  Spend an average of 24 minutes inside What is the average number of customers inside the parlor?

27. Answer We compute X and apply Little’s Law

28. Answer We compute X and apply Little’s Law  X = 120/6 = 20 customers/hour  R = 24 minutes = 0.4 hours  N = XR = 8 customers

29. If you did not get it The 120 customers sent a total of 120×24 customer×minutes or 48 customer×hours in the parlor  48 customer×hours/6 hours = 8 customers Same as having 8 customers spending six hours each inside the parlor

30. Network of servers (I) Arrivals Departures Open network

31. Network of servers (II) Arrivals Departures Closed network

32. Operational Quantities Keep same quantities as before but add indices  0 for whole system  k for individual servers Two changes  We never care about the utilization of the whole system  We add number of visits V k of each server

33. Operational quantities Over the observation period, we measure  C = the number of job completions  C k = the number of tasks completed by device k We define  X 0 = C/T = the system throughput  X k = C k /T = the output rate at server k  V k = C k /C = the visit count at server k

34. Important relationships C k = V k C  Since each job requires V k visits, there are V k more server completions than job completions X k = V k X 0  Same property applies to throughputs

35. System response time (I) We define  Nbar = average number of jobs in the system  nbar i = average number of jobs at device i Nbar = Σ i nbar i

36. System response time (II) Applying Little’s law, we have R = Nbar/X 0 and nbar i = R i X i = R i V i X 0 Hence R = Σ i V i R i

37. Note This result is trivial  The total time spent by a job in the system is the sum of the times spent at each server This includes the time spent waiting in the server queues

38. Problem 1 A job requires  100 ms of CPU time  9 disk accesses Each disk access takes 7 ms We want  V CPU and S CPU

39. Answer We now that jobs get CPU first and last  V CPU = 10 Then  S CPU = 100/10 =10s

40. Bottleneck analysis (I) A system has one CPU and one disk drive It processes transactions such that  V CPU = 12 and S CPU = 5ms  V Disk = 11 and S DISK = 8ms What is the maximum system throughput?

41. Bottleneck analysis (II) We compute first the maximum device throughputs Maximum X CPU = 1/0.005 = 200 requests/s Maximum X disk = 1/0.008 = 125 requests/s Since X i = V i X 0  Maximum throughput compatible with CPU workload is 200/12 = 16.7 transactions/s  Maximum throughput compatible with disk workload is 125/11 = 11.4 transactions/s

42. Bottleneck analysis (III) The disk is this the bottleneck  It has highest V i S i product Identifying feature of any bottleneck device Increasing the system throughput might require  Sharing disk requests with a second disk  Increasing the efficiency of the system I/O buffer

43. Problem 2 In the previous example, which device was the bottleneck? What would be the throughput of the system if the bottleneck utilization was 80%?

44. Answer We compare  V CPU S CPU  V disk S disk

45. Answer We compare  V CPU S CPU = 100ms  V disk S disk = 9×7 = 63 ms The CPU is the bottleneck

46. Answer If the bottleneck was operating at 100% utilization,  It could process one job each V CPU S CPU time units  Or 1/(V CPU S CPU ) job per time unit At U CPU utilization,  It will process U CPU /(V CPU S CPU ) job per time unit

47. Answer X 0 = U CPU /(V CPU S CPU ) = 0.80/0.10 seconds  8 jobs/second

48. Systems with terminals M Terminals Whole system

49. Interactive response time formula We have  M terminals  Think time Z between the completion of a job and the submission of the next job Applying Little’s law to the whole system M = (R + Z ) X 0 then R = M/X 0 – Z Very Important

50. Problem 3 We have  M = 50 users  Z = 20 s  X 0 = 2 transactions/s What is the system response time?

51. Answer We apply R = M/X 0 – Z

52. Answer We apply R = M/X 0 – Z and obtain R = 50/2 – 20 = 5 seconds

53. Problem 4 A system  Processes 5 transactions/seconds  Has 60 users  Achieves a response time of 4 seconds What is the think time?

54. Answer We apply R = M/X 0 – Z,  Z = M/X 0 – R

55. Answer We apply R = M/X 0 – Z,  Z = M/X 0 – R = 60/5 – 4 = 8 seconds

56. Problem 5 We have  M = 50 users  Z = 20 s  R = 4 s What is the system throughput?

57. Answer From R = M/X 0 – Z, we have X 0 = (R + Z)/M Hence X 0 = (20 + 4)/50 = 0.48 tasks/s

58. Problem 6 A system  Can process up to 4 transactions/second  Has 60 users  User think time is 12 seconds Can the system achieve a response time of 2 seconds?

59. Answer Applying R = M/X 0 – Z, we compute a lower bound for the response time  R min = M/X 0,max – Z

60. Answer Applying R = M/X 0 – Z, we compute a lower bound for the response time  R min = M/X 0,max – Z = 60/4 – 12 = 3 seconds Answer is no

61. Problem 7 Compute the response time of a system knowing the following parameters  M = 50 users  Z = 15 s  V CPU S CPU = 200ms  U CPU = 50%

62. Answer Since X k = U k /S k and X k = V k X 0, X 0 = U k /(V k S k ) The response time is then given by R = M/X 0 – Z

63. Answer Let us compute first the throughput X 0  Applying X 0 = U k /(V k S k ) X 0 = 0.50/0.200 = 2.5 interactions/s The response time is then R = M/X 0 – Z = 50/2.5 – 15 = 5 s

64. Simulation Case Studies

65. A simple reminder If interarrival times are  Independent identically distributed (i. i. d.)  According to an exponential law then the probability of having exactly n arrivals during a fixed interval is distributed according to a Poisson law

66. Explanation (II) Assume that  The probability of one arrival during a small interval  t is  t  The probability of two arrivals during the same small time interval is negligible tt tt tt tt tt tt

67. Explanation (I) The probability of having exactly k arrivals during n slots is What would happen if the number of time intervals goes to infinity while their total duration T = n  t remains constant

68. Explanation (III) We rewrite the previous expression as and compute separately the limits of its four factors

69. Explanation (IV)

70. Explanation (V) We obtain the Poisson distribution The probability that there are no arrivals in the same time interval T (or in any time interval T ) is

71. Explanation (VI) This last expression is the probability that the time interval between two consecutive arrivals is greater than T The probability that the time interval between two consecutive arrivals is equal or lesser than T is which is the cdf of the exponential distribution

72. A final observation Use the Poisson distribution to generate number of arrivals during a time interval Use the exponential distribution to generate interarrival times

73. Linear Regression

74. Most important point Compute a regression line Compute regression coefficient

75. Example

76. Linear Regression We have  one independent variable  One dependent variable We must find Y =  +  X minimizing the sum of squares of errors  i (y i -  -  x i ) 2

77. Formulas

78. Calculations (I)

79. Calculations (II)

80. Outcome

81. More notations

82. More notations (II) Solution can be rewritten

83. Coefficient of correlation r = 1 would indicate a perfect fit r = 0 would indicate no linear dependency

84. Calculations