1. Discussant
Webster West

2. My experience
Taught a simulation based intro stat course at NCSU over several semesters.
Collected large amounts of data on how students learn via simulation
No strong evidence that students learn more when I teach with simulation
Tactile simulations do seem to improve the learning process a bit.

3. Comments/Questions for Karsten
In many ways, this is the most well designed study with the most randomization.
Odd that the only improvements with simulation came for confidence intervals.
Very disappointing that the P-value is still elusive even with simulation.
How does the use of the ARTIST instrument make this study compare to the others that are CAOS based?

4. Comments/Questions for Beth
This study appears to do the most with covariate information.
Pre/post design is very nice.
There still appears to be some nasty lurking variables primarily related to instructor/curriculum.
CAOS type questions are probably much less likely to be used in traditional courses.
Why aren’t we seeing more improvements at the end of the semester?

5. Comments/Questions for Nathan
The focus on background ability is interesting.
Pre/post is a nice design.
There is a lack of randomization and accounting for confounding factors – instructor, etc.
CAOS type questions are probably much less likely to be used in traditional courses.
Why aren’t we seeing more improvements at the end of the semester?

6. Comments/Questions for Bob
Lack of pre/post data and other covariates (like instructor info) makes interpreting the results very difficult.
The instructor effect is very clear in that the Minnesota student performed much better than others.
CAOS type questions are probably much less likely to be used in traditional courses.
Why aren’t we seeing more improvements at the end of the semester (outside of Minnesota)?

7. General Comments What are we missing with simulation?
Standardization is very important.
Inference with summary statistics
One sample hypothesis testing problems are not natural with simulation.
Why are we so concerned with inference?
Random samples are almost impossible to come by
Basic inference is often not very interesting with even moderately sized data sets.
Can’t we do something better?