1. Center for Biofilm Engineering Marty Hamilton Professor Emeritus of Statistics Montana State University Statistical design & analysis for assessing the efficacy of instructional modules CS 580 April 24, 2006

2. Why Statistics? Provide convincing results Improve communication “...I do not mean to suggest that computers eliminate stupidity---they may in fact encourage it.” Robert P. Abelson, in Statistics as Principled Argument (cited on Rocky Ross’s CS 580 home page)

3. What is Statistics? Data Design Uncertainty assessment

4. Statistical Thinking Data Design Uncertainty assessment

5. Data:Choosing the quantity to measure Reliable test of knowledge Quantitative response

6. Statistical thinking Data Design Uncertainty assessment

7. After-treatment score A student used the modules, then scored 80% on the test Conclusion: modules have high efficacy

8. Data:Choosing the quantity to measure Reliable tests of knowledge: before-treatment test after-treatment test Quantitative response: difference in test scores, after-treatment minus before-treatment

9. After-treatment score High Low Test score After

10. Before- and after-treatment scores High Low After Before Test score Response

11. Difference between before- and after-treatment scores A student used the modules, then scored 50 points higher on the after- treatment test than on the before treatment test (Response = 50). Conclusion: modules have high efficacy

12. Anticipating criticism: “natural” improvement High Low After Before Test score without the treatment Response

13. Anticipating criticism Before/after observations for just the “treated” student may not accurately represent the treatment effect May need treated and untreated students (i.e., a control)

14. Control or comparison The control can be either a negative control or positive control A student taking a conventional classroom lecture/recitation course would provide a positive control or comparison (placebo) (best conventional)

15. Difference scores for each of 12 students, 6 per group 100 0 Treated group Control group Difference (after – before) Of practical importance?

16. Study design Before and after test scores for each student in both the treated and control groups

17. Good study design Control or comparison Replication Randomization Anticipate criticism

18. Data: 20 students per group (randomly assigned?) TreatmentResponse C-28.5096 C34.7186 C-3.3184 C-13.9297 C-5.7949 C29.0260 C15.4682 C29.1025 C-10.8522 C-18.7876 C-3.1457 C5.4531 C-9.3185 C1.2575 C-11.5470 C-17.6932 C5.5314 C6.7628 C-10.8001 C18.3930 TreatmentResponse T53.4115 T75.9697 T8.3348 T33.3584 T42.5355 T58.2345 T47.9143 T58.6826 T48.3604 T68.2412 T91.1052 T42.8328 T48.9096 T67.1174 T39.2733 T68.9961 T52.2039 T39.2210 T31.1658 T36.4764

19. Analysis via Minitab 14.Minitab: FirstStudy_CS580.MTW Show data layout... matrix Stat > Basic Statistics > Display Descriptive Statistics... Ask for individual value plot Stat > Basic Statistics > 2 Sample t... Minitab output Two-Sample T-Test and CI: Response, Treatment Two-sample T for Response Treatment N Mean StDev SE Mean C 20 0.6 17.4 3.9 T 20 50.6 18.4 4.1 Difference = mu (C) - mu (T) Estimate for difference: -50.0164 95% CI for difference: (-61.4656, -38.5672) T-Test of difference = 0 (vs not =): T-Value = -8.84 P-Value = 0.000 DF = 38 Both use Pooled StDev = 17.8846 Null hypothesis: true mean response for Treatment = true mean response for Control Conclusions: 1. Reject the null hypothesis because it is discredited by the data (p-value < 0.001) 2. 95% confident that the treatment mean response is between 38.6 and 61.5 larger than the true control mean response 3. Is this efficacy repeatable?

20. Analysis via Minitab 14 (more)

21. Minitab: SixStudies_CS580.MTW Show data layout... matrix Stat > Tables > Descriptive Statistics Minitab output Tabulated statistics: Replicate, Treatment Rows: Replicate Columns: Treatment C T All 1 0.60 50.62 25.61 20 20 40 2 0.07 62.94 31.50 20 20 40 3 5.09 51.46 28.27 20 20 40 4 13.29 58.99 36.14 20 20 40 5 6.85 41.45 24.15 20 20 40 6 16.05 51.59 33.82 20 20 40 All 6.99 52.84 29.92 120 120 240 Cell Contents: Response: Mean Count

22. Analysis via Minitab 14 (more)

23. Stat > ANOVA > General Linear Model... Minitab output General Linear Model: Response versus Treatment, Replicate Factor Type Levels Values Treatment fixed 2 C, T Replicate(Treatment) random 12 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6 Analysis of Variance for Response, using Adjusted SS for Tests Source DF Seq SS Adj SS Adj MS F P Treatment 1 126120 126120 126120 128.16 0.000 Replicate(Treatment) 10 9840 9840 984 2.53 0.007 Error 228 88786 88786 389 Total 239 224746 S = 19.7335 Variance Components, using Adjusted SS Estimated Source Value Replicate(Treatment) 29.73 Variance among replicate studies Error 389.41 Variance among students in same study and treatment ---------added by Marty ---------- Total variance 419.14 Repeatability Standard Deviation = 20.5 (single student) Repeatability Standard Deviation = 9.9 (mean of 20 treated students minus mean of 20 control students) Stat > Basic Statistics > Normality Test... of residuals provides an evaluation of key statistical assumption underlying the ANOVA

24. Analysis via Minitab 14 (more) Data copied from Tables output and pasted into the worksheet: Rep CntrlMean TrtMean Mean (Treatment minus Control) 1 0.60 50.62 50.02 2 0.07 62.94 62.87 3 5.09 51.46 46.37 4 13.29 58.99 45.70 5 6.85 41.45 34.60 6 16.05 51.59 35.54 Stat > Basic Statistics > 1 sample t... analysis of 6 Means Conclusions: 1. Reject the null hypothesis because it is discredited by the data (p-value < 0.001) 2. Estimated difference in mean responses = 45.9 3. 95% confident that the treatment mean response is between 36.9 and 54.9 larger than the true control mean response 4. 95% confident that the treatment mean response is at least 38.6 larger than the true control mean response 5. The efficacy measure is repeatable Note: this straightforward analysis of the six means, one mean for each of the 6 repeated studies, using a 1-sample t-test provides nearly the same results as does the ANOVA variance component analysis approach.

25. Trade-offs: What is the main source of variability? It is often more important to repeat the study than to expend time and materials finding a precise efficacy estimate for a single study.

26. Fin