On-line auto-graded assignments in economics: A test of their effectiveness Brendan Kennelly 1, John Considine 2 and Darragh Flannery 1 1 Department of Economics, NUI Galway 2 Department of Economics, UCC

Structure of Presentation Background Experiment Descriptive statistics Econometric analysis Discussion

Background: What is Aplia? Designed by Paul Romer in 2000 Course management tool Facilitates learning and assessment –Using diagrams –Practice Assignment –Instant Feedback Income doubles for this consumer. Place new budget constraint on diagram.

Background: Previous Work Technology –Teaching/Learning –Cost to Instructor Aplia –Collins et al (2008) –ODea and Ring (2008) Aplia in Nat. Univ. of Ireland, Galway –Kennelly and Duffy (DEE 2007) All Assignments on Aplia Evaluated By Student Survey

A New Experiment Brendan Kennelly and David Duffy Poster Presentation to Developments in Economic Education Conference. Evaluating the Use of Aplia as a Substitute to Alternative First Year Tutorials Using Student Survey Darragh Flannery (econometrics) and John Considine (student opinion) replace David Duffy Aplia being used as a complement to tutorials for Second Year students Evaluate by Student Survey and Focus Groups Evaluate by examining grade outcomes

A New Experiment 390 students took managerial economics Analysis mainly about 254 Commerce Students 10 assignments; all did 1 and 2 on Aplia Thereafter ¼ did a paper assignment instead of Aplia Exam in 8 sections corresponding to the last 8 assignments Students had to do 6 sections in the exam

Assignment grade based on best 8 out of 10 Examination grade based on 6 question out of 8

VariableMeanStandard Deviation MinimumMaximumScale Overall Exam Score Assignment Assignment Assignment Assignment Assignment Assignment Assignment Assignment Overall Assignment Mark CAO points st Year Economics Result Overall Course Total Summary Statistics

Descriptive Statistics Table 2: completion rate was lower for the written assignments than for the Aplia ones. Table 3: no consistent pattern in the marks across assignment format (Aplia or written) Table 4: no consistent pattern in the average mark per examination section by each group Table 5: average mark varied considerably across exam section

A1A2A3A4A5A6A7A Table 2: Percentage of Group Completing Assignment

Econometric analysis: overall exam performance VariableCoefficientP-value Constant Overall Assignment Mark1.40 CAO Points.20 Results in 1 st Year Economics1.850 Gender VariableCoefficientP-value Constant No. of Assignments Complete8.80 CAO Points.230 Results in 1 st Year Economics2.20 Gender Observations254

Overall exam performance The results are in line with expectations. We replaced overall assignment mark with a variable that measured the number of assignments attempted. This variable is also significant and has a very large coefficient. However there is relatively little variation in this variable.

Econometric analysis: section specific regressions

Section specific regressions Aplia/paper dummy is statistically significant only in Section 2. In that case students who did the assignment by Aplia performed somewhat better in that section of the exam. The number of marks that a student received in an assignment is statistically significant with a positive sign in the regressions on Section 3 and Section 8 and is weakly significant in the regression on Section 1.

Panel analysis The variation across students and across sections in our dependent variable meant that our data could be analysed using panel data techniques Pooled data Fixed effects Random effects

Econometric analysis: Pooled Data VariableCoefficientP-value Constant Assignment Mark.090 Aplia/Paper Dummy Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section CAO Points.0330 Results in 1 st Year Economics.350 Gender Observations1343

Econometric analysis: fixed effects VariableCoefficientP-value Constant Assignment Mark Aplia/Paper Dummy Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section Rho.43 Observations1343

Econometric analysis: Random Effects VariableCoefficientP-value Constant Assignment Mark Aplia/Paper Dummy Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section Dummy for Section CAO Points.0340 Results in 1 st Year Economics.360 Gender Rho.136 Observations1343

Discussion: Results Very little evidence that Aplia v Written format of the assignment matters Little evidence of a direct link between performance in an assignment and mark on the corresponding section in the exam. If there is no difference in learning outcome then –Cost (student & instructor) becomes important –Students preferences become important

Discussion: Issues Where might learning from Aplia take place? –Between practice assignment and actual assignments? –Incrementally so that ALL students do a better examination – all have used Aplia 6 times? Does the examination test learning from Aplia? Is the experiment testing the benefits of technology and/or assignments rather than Aplia? Is there cheating and/or cooperative learning?

Discussion: Future Work More analysis of existing data –Including two stage estimation Change Experiment –Do not allow practice assignments for some of assignments Compare with other assignments Compare with exact same assignment Is Aplia fit-for-purpose? –What purpose?