1. Improving Reading Compliance and Quiz Scores through the Use of Reading Guides Trent W. Maurer & Judith Longfield

2. What Does the Literature Say? Reading Compliance Compliance has declined over past 30 years—more than 80% to less than 20% (Burchfield & Sappington, 2000) Even in textbook-reliant intro courses, students read less than 1/3 of assigned pages (Gurung & Martin, 2011) Compliance significantly predicts exam scores & final grades (Sappington et al., 2002)

3. Reading Quizzes Graded quizzes provide external incentive for doing reading (Ruscio, 2001) Short quizzes at start of class are effective when paired with prompt feedback ( Connor-Greene, 2000) Reduce procrastination & massed practice (Maki & Maki, 2000) Provide feedback to students on effectiveness of their studying (Rosenthal & McKnight, 1996) Literature Review (cont.)

4. Reading Guides Instructor determines learning outcomes of assigned reading text & creates structured series of questions that o guides students through text o assists them in determining meaning o helps them achieve basic comprehension & vocabulary (Horning, 2007) Literature Review (cont.)

5. Reading Guides (cont.) Help students get more out of readings (Herber, 1978) “Model how to select, decide, & focus upon what textbook material is important for learning” (Helms & Helms, 2010, p. 109) Perceived by students to be helpful in learning objectives & preparing for graded assessments (Helms & Helms, 2010) Students who complete them score higher on graded assessments (Meiss, 1983) Literature Review (cont.)

6. What Was the Impetus? Administratively imposed curriculum redesign on key foundational intro course To meet new learning objectives, class had to be “flipped.” Daily reading quizzes became a necessity o Students bombing daily reading quizzes (M = 51%) o Students claimed they were doing the readings Could reading guides help? Could additional online quizzes help?

7. Fall 2010 Course Grades

8. What Was the Hypothesis? Student performance on daily reading quizzes would improve in a linear fashion as the following components were added to the course (by section): a. Reading guides b. Online practice quizzes c. Online graded quizzes d. Both online practice quizzes & online graded quizzes

9. What Methods Were Used? Participants 5 sections Intro Child Development students (N = 290) Control: 64 RG only: 79 RG + Practice Quiz: 78 RG + Graded Quiz: 39 RG + Both Quizzes: 40 No demographic data collected

10. Materials Instructor-designed guides for 20 assigned readings Daily 5-item quizzes—1/2 chapter each Daily self-report survey (except Control section): o Reading completed o RG completed o Hours studied (sections 3, 4, & 5 only) 20 online 5-item practice quizzes “due” before class (sections 3 & 5 only) 20 online 5-item graded quizzes due before class (sections 4 & 5 only) Methodology (cont.)

11. End of Course Survey (except Control section): Average readings completed Average RG completed Average hours studied 4 items evaluating RGs Methodology (cont.)

12. Procedure On 2 nd day, instructor explained how to use RGs 3 rd day assigned readings started RGs available through CMS from start of course Last day—students completed end of course survey Methodology (cont.)

13. Procedure (cont.) 20 daily reading quizzes at start of class—quizzes preceded by self-report survey Groups 3 & 5—online practice quizzes opened after class, remained open until 30 min. before start of next class Groups 4 & 5—online graded quizzes opened after class, remained open until 30 min. before start of next class Methodology (cont.)

14. What Were the Results? Daily Reading Quiz Averages by Section Section Control (N = 64)RG Only (N = 79)RG + Practice (N = 78)RG + Graded (N = 39)RG + Both (N = 40) MSEM M M M 51.22%1.87%63.80%1.69%67.78%1.81%48.00%2.40%54.70%2.37%

15. ANOVA F (4, 285) = 18.19, p <.001, partial η 2 =.20 Post hoc analyses using Least Significant Differences, effect sizes using Cohen’s d: o Control < RG only (0.85), RG + Practice (1.11) o RG only > Control, RG + Graded (1.06), RG + Both (0.61) o RG + Practice > Control, RG + Graded (1.33), RG + Both (0.88) o RG + Graded < RG only, RG + Practice, RG + Both (0.45) o RG + Both RG + Graded (??) Results (cont.)

16. ANCOVA (excludes Control section) Covariates from end of course survey: o Self-reported reading compliance (ns); r =.28, p <.001 o Self-reported RG completion, F (1, 178) = 22.62, p <.001, partial η 2 =.11; r =.41, p <.001 o Self-reported study time (ns); r =.07, ns Main effect for section, F (3, 178) = 23.63, p <.001, partial η 2 =.29

17. Results (cont.) Estimated marginal means assuming approximately: 75% reading completion 75% RG completion 3-6 hours weekly study time Section RG Only (N = 79)RG + Practice (N = 78)RG + Graded (N = 39)RG + Both (N = 40) MSEM M M 65.49%1.61%71.32%1.63%50.99%2.14%56.29%2.09%

18. Daily vs. End of Course Reports Note. Reading and RG for sections 2, 3, 4, and 5 only. Study Hours for sections 3, 4, and 5 only. Report DailyEnd of Course VariableMSEM tdfpCohen’s drp Reading3.63.064.05.067.36190.0000.51.57.000 RG3.64.084.16.079.38188.0000.50.71.000 Study Hours 2.78.063.17.086.16125.0000.49.59.000

19. Results (cont.) ANCOVA (excludes Control section) Covariates from daily survey: o Self-reported reading compliance, F (1, 220) = 23.72, p <.001, partial η 2 =.10, r =.59, p <.001 o Self-reported RG completion, F (1, 220) = 27.36, p <.001, partial η 2 =.11, r =.64, p <.001 Main effect for section, F (3, 220) = 33.72, p <.001, partial η 2 =.32

20. Results (cont.) Estimated marginal means assuming approximately: 50-75% reading completion 50-75% RG completion Section RG Only (N = 79)RG + Practice (N = 78)RG + Graded (N = 39)RG + Both (N = 40) MSEM M M 63.87%1.17%67.76%1.27%51.90%1.69%50.79%1.71%

21. Results (cont.) ANCOVA (excludes Control and RG Only sections) Covariates from daily survey: o Self-reported reading compliance, F (1, 141) = 10.95, p =.001, partial η 2 =.07 o Self-reported RG completion, F (1, 141) = 16.53, p <.001, partial η 2 =.11 o Self-reported study time (ns); r =.41, p <.001 Main effect for section, F (2, 141) = 38.98, p <.001, partial η 2 =.36

22. Results (cont.) Estimated marginal means assuming approximately: 50-75% reading completion 50-75% RG completion 2 hours study time/quiz Section RG + Practice (N = 78)RG + Graded (N = 39)RG + Both (N = 40) MSEM M 67.95%1.36%51.82%1.81%50.69%1.84%

23. Overall Quiz Means across Adjustments

24. Results (cont.) End of Course Survey Analyses (MANOVA), excluding Control section 7 survey questions (significantly intercorrelated) as DVs Section was significant, Wilks’ Lamba =.73, F (21, 474.34) = 2.57, p <.001, partial η 2 =.10 Significant univariate models only for: o Self-reported reading compliance, F (3, 171) = 4.15, p =.007, partial η 2 =.07 o Perception of RGs as helpful in preparing for exams, F (3, 171) = 3.57, p =.015, partial η 2 =.06 No differences for RG completion, Study Hours, helpfulness of RGs during reading, helpfulness of RGs preparing for class activities, helpfulness of RGs preparing for quizzes

25. End of Course Survey Means by Section Section RG Only RG + Practice RG + GradedRG + Both ItemMSEM M M Reading4.40 a.113.93 b.103.94 b.144.00 b.13 RG4.23.134.31.134.07.164.03.16 Hours Study3.04.113.24.113.36.143.00.14 Helpful—Readings4.38.164.33.164.27.204.30.20 Helpful—Activities4.23.134.47.134.36.174.27.17 Helpful—Quizzes4.51.124.53.124.30.164.59.15 Helpful--Exams4.06 a.124.27 ab.124.46 bc.164.68 c.16 Note. Means within a row with different subscripts significantly different at p <.05.

26. Discussion Student performance on daily reading quizzes improved in a curvilinear fashion across sections o Addition of RGs resulted in roughly 13% raw increase in student quiz scores (25% proportionate) o Addition of practice online quizzes resulted in roughly 5% raw increase beyond impact of RGs (for combined proportionate increase > 1/3)

27. Discussion (cont.) Graded online quizzes, whether alone or in conjunction with practice online quizzes, appeared to fully offset the effect of RGs & return quiz averages to that of Control Section o Additional graded assessments decreased the relative contribution of in class reading quizzes to final grade o May have disincentivized preparing for in class quizzes

28. Discussion (cont.) However, additional analyses revealed that students in the Both Section reported significantly higher daily reading compliance & hours studied than the other sections o Online graded quizzes—because they were open book—may have given students a false sense of confidence in their understanding of the material in a way that the practice quizzes did not

29. Discussion (cont.) The “Sweet Spot” appears to be reading guides in conjunction with online practice quizzes. o RGs give students a way to organize their reading & studying, if they choose to use them. o Practice quizzes give students a way to test their knowledge & receive formative feedback without undermining the importance of in class quizzes for their grades.

30. Limitations One course, one discipline, one instructor, one university Organic design meant partial missing data from 2 sections No way to conclusively establish equivalency between sections in student abilities

31. Future Directions More courses, more disciplines, more instructors, more institutions A priori design to ensure consistency across sections Collect additional data to control for potential pre-existing differences between students (e.g., GPA) Questions?

32. References Burchfield, C. M., & Sappington, J. (2000). Compliance with required reading assignments. Teaching of Psychology, 27, 58-60. Connor-Greene, P. A. (2000). Assessing and promoting student learning: Blurring the line between teaching and testing. Teaching of Psychology, 27, 84-88. Gurung, R. A. R., & Martin, R. C. (2011). Predicting textbook reading: The Textbook Assessment and Usage Scale. Teaching of Psychology, 38, 22-28. Helms, J. W., & Helms, K. T. (2010). Note launchers: Promoting active reading of mathematics textbooks. Journal of College Reading and Learning, 41, 109-119. Herber, H. L. (1978). Teaching reading in content areas (2 nd ed.). Englewood Cliffs, NJ: Prentice-Hall. Horning, A. S. (2007). Reading across the curriculum as the key to student success. Across the Disciplines, 4. Retrieved from http://wac.colostate.edu/atd/articles/horning2007.cfmhttp://wac.colostate.edu/atd/articles/horning2007.cfm Maki, W. S., & Maki, R. H. (2000). Evaluation of a Web-based introductory psychology course: II. Contingency management to increase use of on-line study aids. Behavior Research Methods, Instruments, & Computers, 32, 240-245. Meiss, G. T. (1983). Help your students read their textbooks more profitably. Journalism Educator, 38, 3-10. Rosenthal, G. T., & McKnight, R. (1996). What do introductory psychology students know about their examinations, and when do they know it? Journal of Instructional Psychology, 23, 137-144. Ruscio, J. (2001). Administering quizzes at random to increase students’ reading. Teaching of Psychology, 28, 204-206. Sappington, J., Kinsey, K., & Munsayac, K. (2002). Two studies of reading compliance among college students. Teaching of Psychology, 29, 272-274.