1. More Monte Carlo Methods
and Variance Reduction
Techniques
(for random quadrature)

2. Importance Sampling
Write I as:
for some pdf with support [a,b].
Then

3. Isn’t this accept-reject sampling?
(Answer: No.)
Importance sampling uses all of the draws from the distribution with pdf f and weights them in an appropriate way.
Also, we do not need to find a function that is always “above” the integrand.
(Not to mention that accept-reject sampling was not even used (so far) to evaluate an integral. It was used to draw random variables.

4. We draw n values X1, X2, …, Xn from the distribution with pdf f
and estimate
with

5. Clearly is an unbiased estimator of I:

6. The variance of :

7. The best that we could do is to have
(perfect information!)
Then

8. We could estimate this variance by rewriting it:
Then
(Note: This estimator is biased, but okay for large samples!)

9. Recall that for a random sample Y1, Y2, …, Yn from a distribution with variance
an estimate for is given by
but most people instead learn about the estimator
because it is unbiased for .

10. The estimator we came up with was
This is simply the first (biased) estimator from the last slide. To show this, let
and let be the estimated variance for a single Yi. Then, note that

11. On the other hand, using
we can come up with the unbiased estimator

12. Example:
Importance sampling with exp(rate=1) density
where X~exp(rate=1).

13. Simulation Results: 0.8862269255 Sim 95% CI 1 0.8855383 1.97115x10-6
( , ) 2
x10-6
( , ) 3
x10-6
( , )
(n=100,000)

14. Example:
Importance sampling with density
where X~

15. Simulation Results: 0.8862269255 Sim 95% CI 1 0.8773822 4.227073x10-5
( , ) 2
x10-5
( , ) 3
x10-5
( , )
(n=100,000)

16. Note: The inaccuracy associated with a poor choice of pdf f will become much more pronounced with a smaller sample size!

17. Antithetic Monte Carlo
(a variance reduction technique)
Example:
Consider the integral
(Of course we know that the answer is 1.)
The simple Monte Carlo approach is:

18. So, we simulate X1, X2, …, Xn iid exp(rate=1) rv’s and we compute Sim
95% CI 1
( , ) 2
( , ) 3
( , )
(n=1,000)

19. The antithetic approach is based on the fact that
For X1 and X2 independent,
So, if we can simulate X1 and X2 so that they are negatively correlated, we can get Var(X1+X2) lower than it would be if they were independent!

20. In our example,
can be re-written as
where Xi and Xi’ are negatively correlated exp(rate=1) rv’s.

21. How do we draw two negatively correlated exponentials?
We can draw an exponential rate rv X by
X=F-1(U) where U~unif(0,1)
But U~unif(0,1) implies that (1-U)~unif(0,1).
So, X=F-1(1-U) is also exponential with rate .
These X’s are negatively correlated by monotonicity of F (and hence of F-1).

22. Simulation: Antithetic with 500 uniforms: Sim 95% CI1
( , ) 2
( , ) 3
( , )
(We have cut the CI lengths almost in half!)

23. In general, for estimated by
we need g(x)/f(x) monotone so that negatively correlated X’s produce negatively correlated g(X)/f(X)’s.

24. Control Variate Sampling
(another variance reduction technique)
sometimes we can analytically solve the approximate integral
Try Taylor series.
and then we could Monte Carlo the remaining integral in