Copyright © 2006 Pearson Addison-Wesley. All rights reserved. Lecture 3: Monte Carlo Simulations (Chapter 2.8–2.10)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-2 Review We want an estimator to form a “best guess” of the slope of a line through the origin. We want an estimator that works well across many different samples: low average error, low average absolute error, low squared errors….

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-3 Review (cont.) We have brainstormed 4 “best guesses”:

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-4 Review (cont.) We will compare these estimators in “horseraces” across thousands of computer-generated datasets. We get to specify the underlying relationship between Y and X. We can see how each estimator does.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-5 Review (cont.) We choose all the rules for how our data are created. The underlying rules are the “Data Generating Process” (DGP). We choose to use the Gauss– Markov Assumptions.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-6 Review (cont.) What is our Data Generating Process (DGP)? – E(  i ) = 0 – Var(  i ) =   2 – Cov(  i,  k ) = 0 i ≠ k –X 1, X 2, …, X n are fixed across samples GAUSS–MARKOV ASSUMPTIONS

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-7 How to Build a Race Track… n = ? – How big is each sample?  = ? – What slope are we estimating? Set X 1, X 2, ….., X n – Do it once, or for each sample? Draw  1,  2, …..,  n – Must draw randomly each sample Form Y 1, Y 2, ….., Y n –Y i =  X i +  i Compute all four estimators Save errors, absolute errors, squared errors Do for 10,000 samples and compare

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-8 What Will We Get? We will get precise estimates of: 1.Mean Error of Each Estimator 2.Mean Absolute Error of Each Estimator 3.Mean Squared Error of Each Estimator 4.Distribution of Each Estimator By running different racetracks, we check the robustness of our results.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-9 Agenda for Today Off to the Races! Today we run the simulations and report the results. How well does each estimator do? Does one estimator outperform the others in all settings? Next time, we move from computational methods to analytical methods.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved What Shall We Explore? Mean Error, Mean Absolute Error, Mean Square Error, & the distribution of each  g Would 1,000 samples do well enough? Does sample size matter? – Try n = 6, n = 100

Copyright © 2006 Pearson Addison-Wesley. All rights reserved What Shall We Explore? (cont.) Does normality matter? – For n = 6? For n = 100? Do X ’s matter? – Try low X ’s & high X ’s Does  matter? – Try  = 2 &  = -3 &  = 20

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved EstimatorMean EstimateStandard Deviation  g  g  g  g A First Monte Carlo: n = 6 ;  = 2 ;  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } How close does each estimator come to  = 2? –Results from 10,000 samples

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A First Monte Carlo (cont.) : n = 6 ;  = 2  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } What is the Mean Error? – Results from 1000 samples EstimatorMean Error Standard Deviation of Error  g  g  g  g What do you conclude?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A First Monte Carlo (cont.) : n = 6 ;  = 2  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } Results from 1000 samples Estimator Mean Abs. Error  g  g  g  g What do you conclude?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A First Monte Carlo (cont.) : n = 6 ;  = 2  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } Results from 1000 samples Estimator Mean Sq. Error  g  g  g  g What do you conclude?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A First Monte Carlo Pictures are often very helpful in econometrics. What do the errors look like for each estimator? Let’s look at some histograms.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Figure 2.6 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Normal Disturbances (1 of 2)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Figure 2.6 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Normal Disturbances (2 of 2)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. Figure 2.6 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Normal Disturbances

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Figure 2.7 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Skewed Disturbances (1 of 2)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Figure 2.7 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Skewed Disturbances (2 of 2)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. Figure 2.7 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Skewed Disturbances

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Who’s Winning? Does one of the estimators seem to be doing better than the others? Does the relative performance of the estimators depend on the racetrack? Let’s vary: – The number of samples we draw – The size of each sample – The distribution of the error term – The X ’s – 

Copyright © 2006 Pearson Addison-Wesley. All rights reserved ,000 SAMPLES Estimator Mean Absolute Error  g  g  g  g EstimatorMean Squared Error  g  g  g  g SAMPLES Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g What if We Draw Only 1,000 Samples? What do you conclude?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Histogram of Errors of  g Samples 10,000 Samples Mean Error of  g Mean Error of  g Do We Need 10,000 Samples?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Mean Error of  g Mean Error of  g Histogram of Errors of  g Samples 10,000 Samples Do We Need 10,000 Samples? (cont.)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Mean Error of  g Mean Error of  g Histogram of Mean Error of  g Samples10,000 Samples Do We Need 10,000 Samples? (cont.)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Mean Error of  g Mean Error of  g Histogram of Mean Error of  g Samples10,000 Samples Do We Need 10,000 Samples? (cont.)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved n = 6 Estimator Mean Error Std. Dev.  g  g  g  g n = 100 Estimator Mean Error Std. Dev.  g  g  g  g Does the Sample Size of Each Dataset Matter? n = 6 vs. n = 100 What do you conclude?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Does n Matter? n = 6 vs. n = 100 n = 100 EstimatorMean Absolute Error  g  g  g  g EstimatorMean Squared Error  g  g  g  g n = 6 Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g What do you conclude?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Does Normality of Errors Matter? Our Monte Carlo DGP has assumed  i ~ N( 0,500 2 ) What if we draw the error term from a different distribution? We will consider two distributions, one bimodal and one skewed.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Normal Errors vs. Bimodal Errors: n = 6  ~ Normal Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g  ~ Bimodally Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g

Copyright © 2006 Pearson Addison-Wesley. All rights reserved  ~ Skewed Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g  ~ Normal Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g Normal Errors vs. Skewed Errors: n = 6

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Error of  g 1,  ~ Normal Error of  g 1,  ~ Bimodally Error of  g 1,  ~ Skewed Does Normality Matter? n = 6 Normal Bimodal Skewed

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Error of  g 4,  ~ Normal Error of  g 4,  ~ Bimodally Error of  g 4,  ~ Skewed Does Normality Matter? n = 6 (cont.) Normal Bimodal Skewed

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Skewed Errors The histogram for the errors of the bimodal distribution of  looks approximately normal. The histogram for the errors of the skewed distribution of  does NOT look approximately normal.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Skewed Errors (cont.) What happens to the histogram of errors if we increase the sample size of each dataset from n = 6 to n = 100?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Error of  g 1,  ~ Skewed, n = Error of  g 1,  ~ Skewed, n = Does n Matter for Normality When Disturbances Are Skewed? n = 6 n = 100

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Error of  g 4,  ~ Skewed, n = Error of  g 4,  ~ Skewed, n = Does n Matter for Normality When Disturbances Are Skewed? (cont.) n = 6 n = 100

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Sample Size and Errors The error of our estimator starts to look more normal as we increase n. Later in the class, we will learn about the “Central Limit Theorem.” When we do hypothesis testing, we want to assume the error term is normally distributed. In large samples, this assumption is not so bad.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Does X Matter? So far, X has been either 10 or 20. What happens to our errors if we use a larger X ?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Small X vs. Large X X ’s 10 and 20 Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g X ’s 40 and 50 Estimator Mean Absolute Error  g  g  g  g Estimator Mean Squared Error  g  g  g  g

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Does  Matter? Estimator Mean Squared Error  = 2  g  g  g  g  = -3  g  g  g  g  = 20  g  g  g  g

Copyright © 2006 Pearson Addison-Wesley. All rights reserved A Scorecard

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Review What have we assumed? What is our Data Generating Process (DGP)? – E(  i ) = 0 – Var(  i ) =  2 – Cov(  i,  k ) = 0 i ≠ k –X 1, X 2, …, X n are fixed across samples GAUSS–MARKOV ASSUMPTIONS

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Review (cont.)  g 4 dominates other three estimators  g 2 runs second n matters (bigger n, smaller errors) Biases seem “small” for  g 1,  g 2,  g 4 ? Normal  → normal  g ’s, any n Non-normal  → approximately normal  g ’s for large n → not so normal  g ’s for small n X matters (bigger X ’s, smaller errors)  itself doesn’t matter

Copyright © 2006 Pearson Addison-Wesley. All rights reserved Why Did We Do the Monte Carlos? 1.To emphasize estimators have sampling distributions. 2.To emphasize sampling properties (Mean Error, Mean Absolute Error, Mean Squared Error) are criteria for selecting estimators. 3.To emphasize that estimators’ traits depend on where the data come from.

Copyright © 2006 Pearson Addison-Wesley. All rights reserved What’s Next? We have seen how our estimators do in a number of specific settings that we have simulated computationally. Are these conclusions robust to a broader range of settings? We could run many, many more simulations. Can we find a more general method for exploring the properties of estimators?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved What’s Next? (cont.) We “guessed” four estimators. One of them did very well (  g 4 ) Could there be other estimators that would do even better? Is there a more systematic method for finding good estimators?

Copyright © 2006 Pearson Addison-Wesley. All rights reserved What’s Next? (cont.) Computer simulations let you see what happens in the worlds you design. For broader findings, we turn to mathematics. Next time, we will review some basic mathematical tools. Then we will use these tools to derive general properties of estimators. We can also use these tools to derive the “best” estimators.