Evidence From Principles of Economics Courses Knowledge Retention and Online Course Work Neal H. OlitskySarah B. Cosgrove University of Massachusetts Dartmouth April 12 th, 2013 The 21 st Century Classroom: Online and Blended Learning

The Questions Does the mode of delivery affect student learning? Does the mode of delivery affect knowledge retention? Traditional Face-to-Face (F2F): no online component Web-Enhanced: all class time is F2F, online homework Blended: an integration of traditional F2F classes and online class components which take the place of class time.

Background A shift toward blended or fully online learning Proportion of students taking at least one online course was nearly 1/3 in 2010 (Allen et al. 2012) 79% of public universities offered at least one blended course (Allen et al. 2007) But does this shift lead to better learning outcomes? Nearly 2/3 of professors surveyed believed that learning outcomes in online courses are inferior or somewhat inferior to F2F courses (Allen et al. 2012)

Background Potential BenefitsPotential Costs Increased flexibility More student-teacher interaction Increased student engagement Expand access to educational offerings Reduced operating costs Difficulty with time management Difficulty with new technology Increased faculty time commitment Inadequate support for professional development The literature highlights both benefits and drawbacks of blended learning (Vaughan 2007)

Background How does blending affect performance in economics courses? Few existing studies, typically comparing online to F2F courses Navarro and Shoemaker (2000): online courses performed better. Coates et al. (2004): Reduced learning in online sections. Sizeable selection bias (students select non-randomly into online courses)

Background Three studies look at blended coursework in economics Early studies did not control for selection Brown and Liedholm (2002): principles of micro Terry and Lewer (2003): graduate micro theory Recent study on mode of homework Hernandez-Julian and Peters (2012): principles of micro No significant difference

This Paper To determine the effect of mode on student learning and knowledge retention, we: Focus our attention on two specific learning outcomes Collect assessment data on traditional F2F, web- enhanced, and blended classes Match assessment outcomes to individuals’ college transcripts and demographic data to provide additional controls Estimate the effect of mode using a difference-in- differences model

Empirical Strategy

Data Collection Sample: 7 sections of principles of macro and micro (2 F2F, 3 web-enhanced, 2 blended) in the 2012/2013 academic year (N = 342). Specific Learning Outcomes (SLOs): Compute and compare opportunity costs of different decision-makers to determine the most efficient specialization of production. Discuss, compute, and illustrate graphically the effects of changes in the determinants of demand and supply on the market equilibrium price and quantity.

Data Collection Two different instructors, but identical textbook, quizzes, tests, assignments and schedules for the first 6 weeks. 10 question MC pretest given on the first day of class (not returned to students) Identical 10 questions on first exam Identical 10 question post-test on last day of class

Data Collection (Cont.) Instructor 1Instructor 2 Course Components UsedF2FBlendedWeb- Enhanced Blended Assigned Chapter ReadingYes Online Reading QuizzesNoYesNoYes In-Class Reading QuizzesYesNoYesNo Article on opportunity costs and follow-up discussion board/writing assignment Yes NoYes In-class Practice Problem SetYes Individual Wiki ProjectNoYesNoYes Assignment from online homework management website NoYes A comparison of the course components by mode & instructor

Data Collection (Cont.) Outcome variable: Score on MC questions (out of 10) Control variables: Past Economics Courses Cumulative Credits Cumulative GPA SAT-M/SAT-V Demographics Instructor Year of first enrollment Program of Study

Descriptive Statistics Fall 2012Spring 2013Combined VariableMeanS.D.MeanS.D.MeanS.D. Pretest Exam Post-Test SAT-Math SAT-Verb Cum. GPA Cum. Credits Male White Freshman Business Previous ECO N

Results – Pretest to Exam 1 VariableBlend v. WebS.E.F2F v. WebS.E.F2F v. BlendS.E. Mode x Assess0.052(0.301)0.130(0.361)0.076(0.365) Mode-0.270(0.212)-0.153(0.277)0.206(0.244) Assess1.995***(0.210)1.998***(0.210)2.034***(0.216) SAT Math0.007***(0.001)0.009***(0.001)0.008***(0.001) SAT Verbal0.000(0.001)0.001(0.001)0.001(0.001) Cum GPA0.776***(0.148)0.525**(0.176)0.679***(0.166) Cum Credits-0.014**(0.005)-0.002(0.005)-0.001(0.007) Male0.491**(0.166)0.433*(0.199)0.344(0.190) White-0.124(0.185)-0.150(0.206)-0.269(0.194) Freshman-1.072**(0.384)-1.143**(0.366)-0.847*(0.335) Business0.057(0.171)0.332(0.195)0.070(0.204) Prev. ECO class0.405***(0.098)0.157(0.103)0.620***(0.149) Constant-1.954**(0.751)-2.924***(0.856)-3.293***(0.875) Adj. R-squared N

Results: Blended vs. Web, Post-test VariablePre vs. PostS.E.Exam vs. PostS.E. Mode x Assess-0.601(0.415)-0.005(0.418) Mode-0.243(0.286)-1.003***(0.298) Assess1.861***(0.297)-0.595*(0.309) SAT Math0.005***(0.002)0.010***(0.002) SAT Verbal0.002(0.002)0.002(0.002) Cum GPA0.622***(0.207)1.046***(0.211) Cum Credits-0.009(0.008)-0.012(0.007) Male0.683***(0.214)0.402*(0.229) White-0.314(0.225)-0.147(0.256) Freshman-0.396**(0.484)-0.883*(0.529) Business0.218(0.218)-0.369(0.246) Prev. ECO class0.242**(0.118)0.368***(0.112) Constant-1.941**(1.049) **(1.069) Adj. R-squared N253257

Additional Specifications Quantile regressions: Mode of delivery shows no difference in improvement for students, regardless of their exam scores. No significant effect of mode when sample is divided by Quartiles of SAT math scores Quartiles of cumulative GPA Gender Whether student has taken economics before

Conclusions Student learning was not statistically different regardless of mode. In general, no significant loss of learning associated with blended coursework. The results are also of small magnitude. Based on limited data (more forthcoming), knowledge retention was not statistically different regardless of mode. Student characteristics seem to be much more important.

Results – Pretest to Post-test VariableBlend v. Web EnhancedS.E. DID post-pre-0.576(0.413) SAT Math0.005**(0.002) SAT Verbal0.002(0.001) Cum GPA0.618**(0.209) Cum Credits-0.008(0.007) Male0.669**(0.213) White-0.301(0.224) Business0.234(0.226) Prev. ECO class0.240*(0.118) Constant-1.941(1.048) R-squared0.309 N253

Summary Statistics VariableMean Standard Deviation Pretest Exam Post-test* SAT-Math SAT-Verbal Cum. GPA Cum. Credits Male White Freshman Business Previous ECO class