1. Random number generation Algorithms and Transforms to Univariate Distributions

2. Random number generators Creating numbers that are uniformly distributed on the interval (0,1) have low autocorrelation are unpredictable Could that be achieved in practice?

3. Two basic types of random number generators (RNG): Hardware generators, also called True Random Number Generators (TRNG) – Creates random numbers “physically” – Physical processes (thermal noise, photo electric effects,…) or Mechanic devices (wheels of fortune, tombola machines, dices,…) – Satisfy conditions of independence and uniform distribution, but has limitations with respect to the amount of numbers that can be produced. – See further e.g. http://www.robertnz.net/hwrng.htm

4. Software generators, also called Pseudo Random Number Generators (PRNG) – Create random numbers using a mathematical or cryptographic algorithm – Classic examples: Congruential generators – Modern methods: Blum-Blum-Shub generator, Shift-register- methods, Mersenne Twister, Blowfish cipher – Come as embedded functions in software or can be linked as separate objects to the program code. – The numbers are not truly random; attention must be made to the type of application.

5. Linear Congruential Generators (LCG) Define a sequence {x k } of integers according to where x 0 is chosen as a starting point. “mod m” means that x k is the remainder after division by m  The result is an integer in the interval [0, m – 1] a and c are constants in the interval [0, m) that need to be carefully selected

6. The construction implies that the generated numbers will get into a “loop” with a certain period Example: Let x 0 = a = c = 7 and m = 10  The period is thus 4 in this case

7. Clearly, for a generator to be useful, one condition is that the period is long. How long can it be? As there are only m possible values, the maximum period length must be m. This is named full period. Exercise: Consider the case where a = c =1 What are your findings?

8. A theorem (Knuth, 1981): An LCG has full period m if and only if c is relatively prime to m a – 1 is a multiple of every prime dividing m, i.e. a – 1 = k  p, k  N for every prime p such that m mod p = 0 a – 1 is a multiple of 4, if m is a multiple of 4

9. For LCGs to work on a computer, the period cannot be longer than the word size, i.e. 2 B, where B is the number of bits. LCGs are thus constructed so that the full period coincides with the word size. However, a and c must the be chosen to achieve as much randomness as possible. Guides for this can be found in several publications; the initial standard reference is Knuth D. (1981) The Art of Computer Programming. Volume 2 – Seminumerical Algorithms. 2 nd ed. Addison-Wesley

10. Each generated integer is transformed into the interval [0,1) by dividing by m, i.e. u k = x k / m. Usually, if a zero is obtained, the next value in the sequence is used instead. The so-obtained sequence of real numbers {u k } are for “good” LCGs uniformly distributed over the interval (0,1).

11. The starting value x 0 is called the seed to the generator has often been set by the computer system clock is more and more set by more sophisticated techniques – to prevent predictability of the sequence – to ensure unique seeds (if clock frequency is low compared with the process to be run) can be used to always create the same sequence of pseudo random numbers if e.g. different scenarios should be compared.

12. LCGs as well as more modern generators (Mersenne Twister) can create series of random numbers fully acceptable for most computational statistics studies. When RNGs are to be used e.g. as components in games (roulette, bingo, lotteries, etc.) both for creating tickets and for drawing winning results, care must be taken in order to prevent the random numbers being predictable.

13. Transforms to Univariate Distibutions The “inverse” method for continuous distributions: Let U be a random variable uniformly distributed on (0,1) Let F U be its cumulative distribution function, i.e. The probability density function (pdf) of U is

14. Then

15. Now, let X be a random variable with cumulative distribution function F X. Set where U is uniformly distributed over (0,1) The cumulative distribution function of Y is now as 0  F X (y)  1 and F U (u) = u for 0  u  1  Y has the same probability distribution as X !!

16. Thus, if U is a realization of a random variable, uniformly distributed on (0,1) [usually denoted U  Re (0,1) ] then a realization of a random variable X with cumulative distribution function F X can be obtained by provided F X -1 can be evaluated (analytically or by numerical approximation) The realization U of course comes from a RNG.

17. Some examples Let X be exponentially distributed, i.e. with pdf  for x  0 [F X (x) = 0 för x < 0]

18. To find F X -1 solve for x the equation Thus the transform from U to X becomes

19. Let X follow a symmetric triangular distribution around zero, i.e. the pdf is

20. F X (x) is obtained as 

21. The inverse F X -1 can now be shown to be (The cases y  (0,1) are left out) Thus a Re(0,1) random variable U is transformed into X by Show this as an exercise!

22. Practical considerations with the “inverse” method: 1. When the inverse cumulative distribution can be explicitly derived  No problem! 2. When not  Numerical solution necessary  Usually time-consuming When is 2 the case? Unfortunately quite often, and the most obvious example is that of normally distributed random variables (coming soon)

23. Generation of general uniform deviates A Re(0,1)-distributed random variable U can easily be transformed to a Re(a, b)-distributed random variable X by U can also be transformed to a random variable X with a discrete uniform distribution on the integers (1, …, n ) by where [  ] depicts the integer part. Question and exercise: Why do we need to add “1”? How can U be transformed to a random variable Y with a discrete uniform distribution on the integers (50, 55, 60) ?

24. Generating normally distributed deviates (Two methods will be presented. There exist more of them!) Method 1: (Central limit theorem use) If U 1, U 2, …, U n is a sequence of (assumed) independent Re(0,1)-distributed random numbers, then is approximately normally distributed if n is large enough. As the Re(0,1)-distribution is symmetric around its mean with no heavy tails, the convergence is quite fast.

25. Exercise: What is the mean and standard deviation of X ? How can we use X to further transform to any normal distribution? Why would n = 12 be a good choice?

26. Method 2: (Box & Muller; Marsaglia) Assume U 1 and U 2 are two independent Re(0,1) random variables. Set V 1 = 2  U 1 – 1 and V 2 = 2  U 2 – 1  V 1 and V 2 are independent Re(–1, 1) random variables. Now, let V 1 represent the x-axis coordinate and V 2 the y-axis coordinate of a point within the unit circle. Using polar coordinates

27. Now, and R and  are independent random variables. If R  0, set Can be intuitively seen but also mathematically proven Otherwise, pick two new Re(0,1) deviates U 1 and U 2

28. Use the first representation and set Then R’ and  are also independent and since R varies between 0 and 1 What is now the distribution of R 2 ? As R  Re(0,1) it can be easily found that R 2 also  Re(0,1)

29. Then Further

30. This gives us as the Jacobian is r in the change of variables.  This shows that X 1 and X 2 are independent N(0,1) random variables

31. Generating general discrete distribution random variables For a random variable X distributed over a finite (and small) set of M numbers, divide the interval (0,1) into M intervals with lengths equal to the individual probabilities of the M possible values. If U belongs to interval i let X be equal to value i

32. Special cases 1. The geometric distribution: can be generated by the transform Too see this, figure out for which values of U where x is an integer, and calculate the corresponding probability

33. 2.The Binomial distribution For small n we can use the methods of dividing the interval (0,1) into n sub-intervals For larger values the conjugate relationship with the Beta distribution can be used 3. The Poisson distribution Can be handled by the relationship with the exponential and gamma distributions