Discussant Webster West

My experience Taught a simulation based intro stat course at NCSU over several semesters. Collected large amounts of data on how students learn via simulation No strong evidence that students learn more when I teach with simulation Tactile simulations do seem to improve the learning process a bit.

Comments/Questions for Karsten In many ways, this is the most well designed study with the most randomization. Odd that the only improvements with simulation came for confidence intervals. Very disappointing that the P-value is still elusive even with simulation. How does the use of the ARTIST instrument make this study compare to the others that are CAOS based?

Comments/Questions for Beth This study appears to do the most with covariate information. Pre/post design is very nice. There still appears to be some nasty lurking variables primarily related to instructor/curriculum. CAOS type questions are probably much less likely to be used in traditional courses. Why aren’t we seeing more improvements at the end of the semester?

Comments/Questions for Nathan The focus on background ability is interesting. Pre/post is a nice design. There is a lack of randomization and accounting for confounding factors – instructor, etc. CAOS type questions are probably much less likely to be used in traditional courses. Why aren’t we seeing more improvements at the end of the semester?

Comments/Questions for Bob Lack of pre/post data and other covariates (like instructor info) makes interpreting the results very difficult. The instructor effect is very clear in that the Minnesota student performed much better than others. CAOS type questions are probably much less likely to be used in traditional courses. Why aren’t we seeing more improvements at the end of the semester (outside of Minnesota)?

General Comments What are we missing with simulation? Standardization is very important. Inference with summary statistics One sample hypothesis testing problems are not natural with simulation. Why are we so concerned with inference? Random samples are almost impossible to come by Basic inference is often not very interesting with even moderately sized data sets. Can’t we do something better?