1. Using an Online Course to Support Instruction of Introductory Statistics CAUSE Webinar (8/14/2007) Oded Meyer Dept. of Statistics Carnegie Mellon University

2. Introduction Educational Mission of Funder Educational Mission of Funder (The William and Flora Hewlett Foundation) (The William and Flora Hewlett Foundation) Provide open access to high quality post-secondary education and educational materials to those who otherwise would be excluded due to: Provide open access to high quality post-secondary education and educational materials to those who otherwise would be excluded due to: Geographical constraints Geographical constraints Financial difficulties Financial difficulties Social barriers Social barriers To meet this goal: To meet this goal: A complete stand-alone web-based introductory statistics course. A complete stand-alone web-based introductory statistics course. openly and freely available to individual learners online. openly and freely available to individual learners online.

3. Moving Instruction Out of the Classroom: Challenges Course Organization and Structure Course Organization and Structure Students often view what they learn as a set of isolated facts. Students often view what they learn as a set of isolated facts. Instructor promotes coherence, sets course path. Instructor promotes coherence, sets course path. Online course: high level of scaffolding in structure is needed. Online course: high level of scaffolding in structure is needed. Course is organized around the Big Picture. Course is organized around the Big Picture. Rigid structure throughout material hierarchy. Rigid structure throughout material hierarchy. Smooth conceptual path. Smooth conceptual path.

7. Challenges (cont.) Effective Use of Media Elements Effective Use of Media Elements Course follows well researched principles to minimize cognitive load imposed by the learning design. For example … Course follows well researched principles to minimize cognitive load imposed by the learning design. For example … Best to reinforce information over auditory and visual channels simultaneously. Best to reinforce information over auditory and visual channels simultaneously.

11. Challenges (cont.) Immediate and Targeted Feedback Immediate and Targeted Feedback Studies: immediate feedback  students achieve desired level of performance faster. Studies: immediate feedback  students achieve desired level of performance faster. We needed to compensate for no immediate instructor – students feedback loops. We needed to compensate for no immediate instructor – students feedback loops. Throughout the course immediate and tailored feedback is given. Throughout the course immediate and tailored feedback is given. mini tutors embedded in the material. mini tutors embedded in the material. self assessments activities (Did I get this?) self assessments activities (Did I get this?)

14. Course Evaluation “ Do No Harm ” Study (Fall 2005): Online course vs. traditional course at CMU. Online course vs. traditional course at CMU. Traditional Intro. Stats. Course: Traditional Intro. Stats. Course: Three 50 min. lectures a week. Three 50 min. lectures a week. One lab a week (approx. 1 TA per 10 students). One lab a week (approx. 1 TA per 10 students). Weekly HW assignments. Weekly HW assignments. Text: Intro Practice Stats (Moore & McCabe, 2006). Text: Intro Practice Stats (Moore & McCabe, 2006). Evaluation: three midterms + comprehensive final. Evaluation: three midterms + comprehensive final.

15. Evaluation: First Study (cont.) Sample (online section) : Sample (online section) : Students were invited to participate in “ online section ”. Students were invited to participate in “ online section ”. Of those who volunteered, 20 students were chosen randomly and reasonably resembled the entire class in terms of gender, race and prior exposure to statistics. Of those who volunteered, 20 students were chosen randomly and reasonably resembled the entire class in terms of gender, race and prior exposure to statistics. Requirements: Requirements: go through the course in a specified pace and complete all activities. go through the course in a specified pace and complete all activities. attend a weekly 50 min. meeting for feedback about their learning experience & questions. attend a weekly 50 min. meeting for feedback about their learning experience & questions. Evaluation: three midterms + comprehensive final (matched in level of difficulty to rest of the class). Evaluation: three midterms + comprehensive final (matched in level of difficulty to rest of the class).

16. Evaluation: First Study (cont.) Results: Results: All but 2 students followed schedule (with up to two days of delay). All but 2 students followed schedule (with up to two days of delay). Three instances of clarifications (regression line, sampling distributions, p-value). Three instances of clarifications (regression line, sampling distributions, p-value). Performance : Performance :

17. Evaluation: First Study (cont.) Exam Numerical Measures TraditionalOnline First Exam EDA, Sampling, Design Sample Size Mean Stand. Dev. 20190.178.59 20 88.75 6.23 Second Exam Probability Sample Size Mean Stand. Dev. 20281.8214.252081.4513.82 Third Exam Inference Sample Size Mean Stand. Dev. 20185.8711.912085.1016.80 FinalComprehensive Sample Size Mean Stand. Dev. 20483.5411.062084.7912.23 Performance: Performance:

18. Evaluation: (cont.) Second Study (Spring 2006): Measuring statistical literacy - CAOS Test. Measuring statistical literacy - CAOS Test. Comprehensive Assessment of Outcomes in a first Statistics course) Comprehensive Assessment of Outcomes in a first Statistics course) (delMas, Ooms, Garfield, Chance) 40 multiple choice items 40 multiple choice items Measures statistical literacy & conceptual understanding. Measures statistical literacy & conceptual understanding. Focus on reasoning about variability. Focus on reasoning about variability. 18 expert raters agreed with the statement: 18 expert raters agreed with the statement: “CAOS measures outcomes for which I would be “CAOS measures outcomes for which I would be disappointed if they were not achieved by students who succeed in my statistics courses.”

19. Evaluation: Second Study (cont.) CMU Sample : CMU Sample : 27 students, same selection process as in first study. 27 students, same selection process as in first study. Same course structure and requirements as in first study. Same course structure and requirements as in first study. Students took the CAOS test as a pretest (n=27), and then as a posttest (n=24). Students took the CAOS test as a pretest (n=27), and then as a posttest (n=24). National CAOS Sample: (delMas et al., AERA 2006) National CAOS Sample: (delMas et al., AERA 2006) 488 students, 18 instructors, 16 institutions, 14 states. 488 students, 18 instructors, 16 institutions, 14 states. 2 yr./tech.: 12.5%, 4 yr. college: 41.6%, Univ.: 45.9% 2 yr./tech.: 12.5%, 4 yr. college: 41.6%, Univ.: 45.9% Prerequisite: no math (28.9%), HS algebra (46.1%), Prerequisite: no math (28.9%), HS algebra (46.1%), college algebra (20.7%), calculus (4.3%) college algebra (20.7%), calculus (4.3%)

20. Evaluation: Second Study (cont.)  Three instances of clarifications (correlation, binomial distribution, sampling distributions). n Average % correct Pretest48843.3 Posttest48851.2 Increase: 7.9% [t(487) = 13.8, p <.001] n Average % correct Pretest2455.8 Posttest2466.5 Increase: 11.7% [t(23) = 4.7, p <.001]  Results:  National CAOS Sample:  CMU Sample:

21. Results (cont.) Results (cont.) Measured outcome † of items with less than 50% of students correct on posttest: Measured outcome † of items with less than 50% of students correct on posttest: Understanding of the purpose of randomization in an experiment (29.2%). Understanding of the purpose of randomization in an experiment (29.2%). Misconceptions: reduces sampling error, increase accuracy of results. Misconceptions: reduces sampling error, increase accuracy of results. Understand how sampling error is used to make an informal inference about a sample mean (8.3%). Understand how sampling error is used to make an informal inference about a sample mean (8.3%). Common mistake (62.5%): basing inference on the sample SD, disregarding the sample size. Common mistake (62.5%): basing inference on the sample SD, disregarding the sample size. † as defined in delMas et al. AERA, 2006 Evaluation: Second Study (cont.)

22. Understand how sampling error is used to make an informal inference about a sample mean (8.3%). Understand how sampling error is used to make an informal inference about a sample mean (8.3%). Common mistake (62.5%): basing inference on the sample SD, disregarding the sample size. Common mistake (62.5%): basing inference on the sample SD, disregarding the sample size. Understanding of the factors that allow generalizing sample results to the population (45.8%). Understanding of the factors that allow generalizing sample results to the population (45.8%). Misconception: if the sample is small relative to the population, generalizing results is problematic. Misconception: if the sample is small relative to the population, generalizing results is problematic. Understanding of the logic of significance test when the null hypothesis is rejected (41.7%). Understanding of the logic of significance test when the null hypothesis is rejected (41.7%). Misconception: rejecting the null  null is false. Misconception: rejecting the null  null is false.

23. Results (cont.) Results (cont.) Items with < 50% of CAOS sample correct Items with < 50% of CAOS sample correct and ≥ 50% of CMU sample correct on posttest and ≥ 50% of CMU sample correct on posttest Describing the distribution of a quantitative variable. Describing the distribution of a quantitative variable. Mean>Median  the distribution is most likely skewed left. Mean>Median  the distribution is most likely skewed left. Interpretation of a boxplot. Interpretation of a boxplot. Correctly estimate and compare SD’s for different histograms. Correctly estimate and compare SD’s for different histograms. Correlations does not imply causation. Correlations does not imply causation. Understanding that statistics from small samples vary more than those from large samples. Understanding that statistics from small samples vary more than those from large samples. Understanding of expected patterns in sampling variability Understanding of expected patterns in sampling variability Selecting appropriate sampling distribution for particular population and sample size Selecting appropriate sampling distribution for particular population and sample size Evaluation: Second Study (cont.)

24. Evaluation (cont.) To summarize the results so far… As far as performance and achieving statistical literacy… the online course definitely “does no harm”. As far as performance and achieving statistical literacy… the online course definitely “does no harm”. For some traditionally difficult statistical ideas (in EDA and some aspects of understanding variability) the online course might have a slight edge over traditional courses. For some traditionally difficult statistical ideas (in EDA and some aspects of understanding variability) the online course might have a slight edge over traditional courses. Given that the course was administered almost “stand-alone”, this was quite encouraging. Given that the course was administered almost “stand-alone”, this was quite encouraging.

25. Evaluation: summary (cont.) The CAOS test pin-pointed important statistical ideas that the online course did not succeed in conveying, and revealed which misconceptions need to be rooted out. The CAOS test pin-pointed important statistical ideas that the online course did not succeed in conveying, and revealed which misconceptions need to be rooted out. Students seem to find the course “friendly”. Students seem to find the course “friendly”. All students reported at least some increase in their interest in statistics. All students reported at least some increase in their interest in statistics. 75% Definitely Recommend 75% Definitely Recommend 25% Probably Recommend 25% Probably Recommend 0% Probably not Recommend 0% Probably not Recommend 0% Definitely not Recommend 0% Definitely not Recommend

26. Student Quotes “I really like the way you can learn individually and at your own pace. If I understand something, I can move through it quickly and take more time on challenging things.” "This is so much better than reading a textbook or listening to a lecture! My mind didn’t wander, and I was not bored while doing the lessons. I actually learned something."

27. Evaluation (cont.) Third Study: Accelerated Learning Study (Spring 07) Accelerated online course vs. “ traditional control ”. Accelerated online course vs. “ traditional control ”. Students could choose to register for an accelerated “ online section ” (8 weeks instead of 15 weeks) Students could choose to register for an accelerated “ online section ” (8 weeks instead of 15 weeks) 25 students were selected at random. Those not chosen  “ traditional control ”. 25 students were selected at random. Those not chosen  “ traditional control ”. Requirements: Requirements: Go through the course in an accelerated pace and complete all the activities. Go through the course in an accelerated pace and complete all the activities. Post questions that they wanted addressed in class. Post questions that they wanted addressed in class. Attend two 50 minute meetings a week for “ focused lectures ”, where we went through more examples that targeted topics/issues that students were struggling with. Attend two 50 minute meetings a week for “ focused lectures ”, where we went through more examples that targeted topics/issues that students were struggling with.

28. Evaluation: Third Study (cont.)  Results: n Average % correct Pretest2156 Posttest2173.5 n Pretest2455.8 Posttest2466.5  Online accelerated course:  Online course group (second study): Increase: 11.7% [t(23) = 4.7, p <.001] Increase: 17.5% [t(20) = 6.9, p <.001]

29. Evaluation: Third Study (cont.) n Average % correct Pretest2156 Posttest2173.5 n Pretest4050 Posttest4053  Online accelerated course:  Traditional control: Increase: 3% Increase: 17.5% [t(20) = 6.9, p <.001]

30. OLI students showed significantly greater gains (pre to post) than the Traditional “control” students on the CAOS test. OLI students showed significantly greater gains (pre to post) than the Traditional “control” students on the CAOS test. 17.5% 3%

31. These effects need to be considered in light of the significant difference between groups at pretest (even after our stratified randomized assignment to groups). These effects need to be considered in light of the significant difference between groups at pretest (even after our stratified randomized assignment to groups). 56% 50%

32. Investigating the pretest scores further, there is a significant linear relationship between pretest and posttest score. Investigating the pretest scores further, there is a significant linear relationship between pretest and posttest score. After accounting for the pretest’s predictiveness, (ANCOVA) there is still a significant advantage for OLI students.

33. Summary of third study & final thoughts:  The online students gained much more (on the CAOS test) than did the “traditional controls”.  This is noteworthy given that the OLI students had half a semester to cover a semester’s worth of material.  I believe that the gain in the third study (course + focused lectures format) was better than the gain in the second study (stand-alone format) because the course was developed as a stand- alone course (how ironic…)

34. An issue that needs to be examined is the effect of the accelerated learning on retention (a follow-up study is planned in down-stream courses). An issue that needs to be examined is the effect of the accelerated learning on retention (a follow-up study is planned in down-stream courses). The format of the third study was among the best teaching experiences I’ve had in my 15 years of teaching statistics. The format of the third study was among the best teaching experiences I’ve had in my 15 years of teaching statistics. I strongly believe (and hope, maybe…) that no online course will ever be able to replace an enthusiastic and engaging teacher. However… I strongly believe (and hope, maybe…) that no online course will ever be able to replace an enthusiastic and engaging teacher. However… Having the students engage with material on their own using an online course supplemented by focused lectures is a “winning combination”. Having the students engage with material on their own using an online course supplemented by focused lectures is a “winning combination”.

35. Contact Information: Oded Meyer Oded Meyer meyer@stat.cmu.edu meyer@stat.cmu.edumeyer@stat.cmu.edu To access the course: To access the course: go to: www.cmu.edu/oli/ and follow the link to the statistics course. go to: www.cmu.edu/oli/ and follow the link to the statistics course.www.cmu.edu/oli/