1. Simulation-based inference beyond the introductory course Beth Chance Department of Statistics Cal Poly – San Luis Obispo bchance@calpoly.edu

2. Background  Simulation-based inference has been advocated as a way to get students to think inferentially early and often in the introductory statistics course  Introduction to Statistical Investigation (Tintle et al)  Statistics: Unlocking the Power of Data (Lock et al)  Introductory Statistics with Randomization and Simulation (OpenIntro)  Statistical Thinking: A simulation approach to modeling uncertainty (U of Minn)

3. Background  Example: Does cloud seeding increase rainfall? Examplecloud seeding

4. Background  Students can use simulation-based inference (e.g., randomization tests, bootstrapping) to  Focus on the meaning of p-value and confidence intervals at a conceptual level (“what could happen with my statistic by chance alone?”),  To explore their own questions (e.g, what about the median instead of the mean)  To focus on the entire statistical process as a whole

5. Background  So what comes next?  Do we move to theory-based methods from that point on?  Can we continue to leverage student understanding to ask even richer investigation questions?  What are some of the explorations available to students building on this approach?  Chi-square, ANOVA, Regression

6. Example – Dr. Spock’s Trial  Dr. Benjamin Spock was tried in 1968 for conspiracy to violate the Selective Services Act  His jury did not contain any women  Was his judge biased against women? Judge 1Judge 2Judge 3Judge 4Judge 5Judge 6Judge 7 Total Women on jury list119197118773014986776 Men on jury list235533287149814035112199 Total3547304052261115525972975 Proportions.336.270.291.341.270.144.336

7. Example – Dr. Spock’s Trial  Explore the data  How can we compare these 7 proportions?  Why don’t I just compare Judge 7 to 0.336?  Why don’t I just compare Judge 7 to Judge 4?  Why don’t I just run C(7,2) two-sample z-tests? Judge 1Judge 2Judge 3Judge 4Judge 5Judge 6Judge 7 Total Proportions.336.270.291.341.270.144.336

8. Example - Dr. Spock’s Trial  Explore the data  Can I run one test to consider the equality of all 7 probabilities at once?  We ask students to develop their own statistic (formula) that measures how different these 7 proportions are as a whole Judge 1Judge 2Judge 3Judge 4Judge 5Judge 6Judge 7 Total Proportions.336.270.291.341.270.144.336

9. Example – Dr. Spock’s Trial  Common initial choices of statistic  Largest – Smallest proportion (Max – Min)  Average proportion  Average difference in pairwise proportions  Statistic: Average difference in proportions  Mean Absolute Difference (MAD)  Students can fairly quickly calculate by hand  Observed MAD (success = woman): 0.071 Does choice of success make a difference?

10. Example – Dr. Spock’s Trial  Consider the behavior of your statistic when the null hypothesis is true  Do you expect it to be large or small? Close to zero? Positive or negative? What kinds of values will you consider to be evidence against the null hypothesis?  Simulation:  Random assignment (shuffle the 2975 jurors among the 7 judges with same totals per judge)  Random sampling from binomial process with common probability of success say  = 0.336

11. Example Example – Dr. Spock’s Trial  Strength of evidence: How often would we get a MAD value of 0.071 by chance alone?  Why isn’t this distribution centered at zero?  Why isn’t this distribution symmetric?  What is the most important feature of this distribution to be learning about?  What conclusion can you draw from this p-value?

12. Example – Dr. Spock’s Trial  What about the chi-square test statistic?  With a binary response:  Comparison to overall proportion rather than pairwise differences  Larger values still evidence against null hypothesis  Can also examine individual “z-scores”  Directly see impact of sample size on size of statistic

13. Example – Dr. Spock’s Trial  What about the chi-square test statistic?  Observed statistic: 62.68  Simulation  Strength of evidence does depend on choice of statistic  Some statistics have a nice mathematical model  Under certain conditions

14. ANOVA? (Multiple means)  Can have the exact same conversation comparing multiple means  Choice of statistic: Max – Min, MAD, F-statistic  If you can describe a formula for the statistic, you can create its null distribution  Can help evaluate the effectiveness of a statistic (Is it monotonic with the evidence against the data? Does it reflect sample size? Power …)  What are advantages to standardized statistics?

15. ExampleExample– Height vs. Foot length  Collect class data on height and foot length  Moderate strong linear association with fun outliers to investigate  How accurately can I predict the height from a footprint?  Advantages to using footprint over handprint?

16. Example – Height vs. Foot length  Simulation-based inference  Statistic: slope or correlation coefficient  Simulation: No association = Random assignment of heights to foot lengths

17. Example – Height vs. Foot length  Lines meet at one point  “Bow-tie” pattern  Symmetric, centered at zero  SD approx. 0.338

18. Example – Height vs. Foot length  Can calculate the standardized statistic estimate – null value standard deviation  Theory-based regression analysis

19. Example – Height vs. Foot length  Simulation: No association = Random assignment of heights to foot lengths  Theory-based inference: No linear association in the means of the conditional response distributions at each x, but those conditional distributions are normally distributed with a common variance  2

20. Example – Height vs. Foot length  Change the simulation: Sampling from finite population (assuming some characteristics about the population)

21. Example – Height vs. Foot length  Change the simulation: Sampling from finite population (assuming some characteristics about the population)

22. Example – Height vs. Foot length  Change the simulation: Sampling from finite population (adjusting some characteristics about the population)

23. Example – Height vs. Foot length  Change the simulation: Sampling from finite population (adjusting some characteristics about the population)

24. Big Ideas  Choice of simulation makes a difference  Especially the stronger the association  Are you pooling the results first  Student exploration, development of ideas  Students can explore factors impacting significance and prediction accuracy (derive formulas)  Look at variance of predictors as a screening step (handprint vs. footprint) In fact! Same issue with comparing means and proportions Random shuffling Binomial sampling

25. Summary  Allow students to explore, develop intuition  Choice of statistic (e.g., German tank problem)  How simulate null model  Combine visual with calculation with conceptual  Predict and test  Technology needs to be complement not barrier  Focus on conceptual understanding of advanced topics (e.g., algebra-based second course)  Simulate re-randomization within blocks

26. Questions?  bchance@calpoly.edu  Simulation-Based Inference blog: https://www.causeweb.org/sbi/