Generating Random Variates

Introduction There are various procedures to produce observations (variates) from some desired input distribution (like exponential, gamma, etc.) The algorithm usually depends on desired distribution But all algorithms have the same general form Note the critical importance of a good random-number generator There may be several different algorithms for the desired distribution

Desired Properties Exact: X should have exactly (not approximately) the desired distribution Example of approximate algorithm: Treat Z = U1 + U2 + ... + U12 – 6 as N(0, 1) Mean and variance are correct; it relies on CLT for approximate normality Range is clearly incorrect Efficient: Low storage Fast Efficient regardless of parameter values (robust) Simple: Easy to understand and implement (often tradeoff against efficiency) Requires only U(0, 1) input If possible, one U ® one X (for synchronization in variance reduction)

Generating Random Variates There are five general approaches to generating a univariate random variables from a distribution Inverse transform method Acceptance-rejection method Composition method Convolution method Special properties

Inverse Transform Method The simplest (in principle) and “best” method in some ways Continuous Case Suppose X is continuous with cumulative distribution function (CDF) F(x) = P(X £ x) for all real numbers x that is strictly increasing over all x Step 2 involves solving the equation F(X) = U for X; the solution is written as X = F–1(U) and we must invert the CDF F Inverting F might be easy (exponential), or difficult (normal) in which case numerical methods might be necessary Algorithm: 1. Generate U ~ U(0, 1) using a random number generator 2. Find X such that F(X) = U and return this value of X

Inverse Transform Method Mathematical proof: Assume that F is strictly increasing for all x. For a fixed value x0 P(X £ x0) = P(F–1(U) £ x0) = P(F(F-1(U)) £ F(x0)) = P(U £ F(x0)) = F(x0) Graphical proof: X1 £ x0 if and only if U1 £ F(x0), so P(X1 £ x0) = P(U1 £ F(x0)) = F(x0)

Examples Weibull (α, β) distribution with parameters α > 0 and β > 0 Density function is CDF is Solving U = F(X) for X, we get Since 1 – U ~ U(0, 1) as well, we can replace 1 – U by U to get

Intuition Behind the Inverse Transform Method Weibull (α = 1.5, β = 6)

Intuition Behind the Inverse Transform Method

The Algorithm in Action

Inverse Transform Method Discrete case Suppose X is discrete with cumulative distribution function F(x) = P(X £ x) for all real numbers x and probability mass function p(xi) = P(X = xi), where x1, x2, ... are the possible values X can take Unlike the continuous case, the discrete inverse transform method can always be used for any discrete distribution Graphical proof: From the above picture, P(X = xi) = p(xi) in every case Algorithm: 1. Generate random number U ~ U(0,1) 2. Find the smallest positive integer i such that U ≤ F(xi) 3. Return X = xi

Example Discrete uniform distribution on 1, 2, ..., 100 xi = i, and p(xi) = p(i) = P(X = i) = 0.01 for i = 1, 2, ..., 100

Example “Literal” inverse transform search 1. Generate U ~ U(0,1) 2. If U ≤ 0.01 return X = 1 and stop; else go on 3. If U ≤ 0.02 return X = 2 and stop; else go on 4. If U ≤ 0.03 return X = 3 and stop; else go on . 100. If U ≤ 0.99 return X = 99 and stop; else go on 101. Return X = 100 Equivalently (on a U-for-U basis) 1. Generate U ~ U(0,1) 2. Return X = ë100 Uû + 1

Generalized Inverse Transform Method X = min{x: F(x) ³ U} This is valid for any CDF F(x) Continuous, possibly with flat spots (i.e., not strictly increasing) Discrete Mixed continuous-discrete Problems with the inverse transform method Inverting the CDF may be difficult (numerical methods) It may not be the fastest or simplest approach for a given distribution

Other Implementations of Inverse Transform Approach Sometimes the inverse function is not analytically available but can be computed numerically. In this case, we can still ply the inverse transform approach by using the numerical inverse. Consider the normal distribution, the CDF cannot be written in closed form so the inverse function does not exist in closed form. However, most software enables numerical calculation of the inverse function. Example: NORMINV(p,m,s) in Excel returns F-1X(p) where X is Normal(m,s) and 0<p<1. Therefore, we can generate realizations from X by: NORMINV(RAND() ,m,s) which is equivalent to: F-1X(U) where U is uniform (0,1).

Acceptance-Rejection Method This is used when inverse transform is not directly applicable or is inefficient (like for gamma and beta distributions) It has continuous and discrete versions; we’ll just do the continuous case, discrete case is similar Goal is to generate X with some density function f Specify a function t(x) that majorizes f(x), (t(x) ≥ f(x) for all x)

Acceptance-Rejection Method Then t(x) ≥ 0 for all x, but , so t(x) is not a density Set Define a new density r(x) = t(x)/c for all x Algorithm Generate Y having density r Generate U ~ U(0,1) (Independent of Y in Step 1) If U ≤ f(Y)/t(Y), return X = Y and stop; else, go back to step 1 and try again (Repeat 1-2-3 until acceptance finally occurs in Step 3) Since t majorizes f, f(Y)/t(Y) ≤ 1 so “U ≤ f(Y) / t(Y)” may or may not occur We must be able to generate Y with density r, hopefully this is easy with our choice of t On each pass, P(acceptance) = 1/c, so we want small c = area under t(x), to “fit” down on top of f closely (we want t and thus r to resemble f closely) Tradeoff between ease of generation from r, and closeness of fit to f

Acceptance-Rejection Method Proof: The key point is that we get an X only conditional on acceptance in step 3

Example of Acceptance-Rejection Method Beta (4,3) distribution, density is f(x) = 60 x3(1-x)2 for 0 ≤ x ≤ 1 The maximum occurs at x=0.6, f(0.6)= 2.0736 (exactly), so let t(x) = 2.0736 for 0 ≤ x ≤ 1 Thus, c = 2.0736, and r is the U(0,1) density function Algorithm: Generate Y ~ U(0,1) Generate U ~ U(0,1) independent of Y If U ≤ 60Y3(1-Y)2 /2.0736, return X=Y; else go back to step 1 and try again P(acceptance) in step 3 is 1/2.0736 = 0.48

Intuition

Intuition A different way to look at it Accept Y if U ≤ f(Y)/ t(Y), so plot the pairs (Y, U t(Y)) and accept the Y’s for which the pair is under the f curve

A Closer Fit Higher acceptance probability on a given pass Harder to generate Y with density shaped like t, can perhaps use the composition method

Example fX(x)=3/4(-x2+1) if -1<x<1 and fX(x)=0 otherwise. Majorizing function? 3/4 x -1 1