1. On-line auto-graded assignments in economics: A test of their effectiveness Brendan Kennelly 1, John Considine 2 and Darragh Flannery 1 1 Department of Economics, NUI Galway 2 Department of Economics, UCC

2. Structure of Presentation Background Experiment Descriptive statistics Econometric analysis Discussion

3. Background: What is Aplia? Designed by Paul Romer in 2000 Course management tool Facilitates learning and assessment –Using diagrams –Practice Assignment –Instant Feedback Income doubles for this consumer. Place new budget constraint on diagram.

4. Background: Previous Work Technology –Teaching/Learning –Cost to Instructor Aplia –Collins et al (2008) –ODea and Ring (2008) Aplia in Nat. Univ. of Ireland, Galway –Kennelly and Duffy (DEE 2007) All Assignments on Aplia Evaluated By Student Survey

5. A New Experiment Brendan Kennelly and David Duffy. 2007. Poster Presentation to Developments in Economic Education Conference. Evaluating the Use of Aplia as a Substitute to Alternative First Year Tutorials Using Student Survey Darragh Flannery (econometrics) and John Considine (student opinion) replace David Duffy Aplia being used as a complement to tutorials for Second Year students Evaluate by Student Survey and Focus Groups Evaluate by examining grade outcomes

6. A New Experiment 390 students took managerial economics Analysis mainly about 254 Commerce Students 10 assignments; all did 1 and 2 on Aplia Thereafter ¼ did a paper assignment instead of Aplia Exam in 8 sections corresponding to the last 8 assignments Students had to do 6 sections in the exam

7. Assignment grade based on best 8 out of 10 Examination grade based on 6 question out of 8

8. VariableMeanStandard Deviation MinimumMaximumScale Overall Exam Score116.846.2724.75254.750-300 Assignment 1 30.589.452.2747.72 0-50 Assignment 2 26.5910.463.1250 0-50 Assignment 3 33.318.394.5450 0-50 Assignment 4 32.528.407.1750 0-50 Assignment 5 29.8512.091.1950 0-50 Assignment 6 30.4210.604.1649.30 0-50 Assignment 7 30.1215.192.1750 0-50 Assignment 8 34.2510.424.4150 0-50 Overall Assignment Mark61.9218.486970-100 CAO points438.8537.45390590390-600 1 st Year Economics Result57.8310.7630880-100 Overall Course Total178.7358.9143.5348.750-400 Summary Statistics

9. Descriptive Statistics Table 2: completion rate was lower for the written assignments than for the Aplia ones. Table 3: no consistent pattern in the marks across assignment format (Aplia or written) Table 4: no consistent pattern in the average mark per examination section by each group Table 5: average mark varied considerably across exam section

10. A1A2A3A4A5A6A7A8 0.900.940.950.940.670.940.920.94 0.920.880.930.860.960.740.91 0.930.920.860.850.910.940.800.95 0.930.880.920.750.930.940.920.69 Table 2: Percentage of Group Completing Assignment

11. Econometric analysis: overall exam performance VariableCoefficientP-value Constant-117.430 Overall Assignment Mark1.40 CAO Points.20 Results in 1 st Year Economics1.850 Gender.28.94 VariableCoefficientP-value Constant-158.850 No. of Assignments Complete8.80 CAO Points.230 Results in 1 st Year Economics2.20 Gender.33.94 Observations254

12. Overall exam performance The results are in line with expectations. We replaced overall assignment mark with a variable that measured the number of assignments attempted. This variable is also significant and has a very large coefficient. However there is relatively little variation in this variable.

13. Econometric analysis: section specific regressions

15. Section specific regressions Aplia/paper dummy is statistically significant only in Section 2. In that case students who did the assignment by Aplia performed somewhat better in that section of the exam. The number of marks that a student received in an assignment is statistically significant with a positive sign in the regressions on Section 3 and Section 8 and is weakly significant in the regression on Section 1.

16. Panel analysis The variation across students and across sections in our dependent variable meant that our data could be analysed using panel data techniques Pooled data Fixed effects Random effects

17. Econometric analysis: Pooled Data VariableCoefficientP-value Constant-16.90 Assignment Mark.090 Aplia/Paper Dummy.66.288 Dummy for Section 23.00 Dummy for Section 3-1.57.1 Dummy for Section 43.850 Dummy for Section 5-7.60 Dummy for Section 6-3.60 Dummy for Section 7-4.30 Dummy for Section 8-3.030 CAO Points.0330 Results in 1 st Year Economics.350 Gender.23.65 Observations1343

18. Econometric analysis: fixed effects VariableCoefficientP-value Constant21.560 Assignment Mark.009.766 Aplia/Paper Dummy.570.334 Dummy for Section 23.10 Dummy for Section 3-1.88.04 Dummy for Section 43.22.001 Dummy for Section 5-7.250 Dummy for Section 6-4.110 Dummy for Section 7-5.010 Dummy for Section 8-3.10 Rho.43 Observations1343

19. Econometric analysis: Random Effects VariableCoefficientP-value Constant-17.10 Assignment Mark.07.011 Aplia/Paper Dummy.64.274 Dummy for Section 23.130 Dummy for Section 3-1.65.07 Dummy for Section 43.610 Dummy for Section 5-7.520 Dummy for Section 6-3.850 Dummy for Section 7-4.60 Dummy for Section 8-3.10 CAO Points.0340 Results in 1 st Year Economics.360 Gender.18.77 Rho.136 Observations1343

20. Discussion: Results Very little evidence that Aplia v Written format of the assignment matters Little evidence of a direct link between performance in an assignment and mark on the corresponding section in the exam. If there is no difference in learning outcome then –Cost (student & instructor) becomes important –Students preferences become important

21. Discussion: Issues Where might learning from Aplia take place? –Between practice assignment and actual assignments? –Incrementally so that ALL students do a better examination – all have used Aplia 6 times? Does the examination test learning from Aplia? Is the experiment testing the benefits of technology and/or assignments rather than Aplia? Is there cheating and/or cooperative learning?

22. Discussion: Future Work More analysis of existing data –Including two stage estimation Change Experiment –Do not allow practice assignments for some of 2009-10 assignments Compare with other 2009-10 assignments Compare with exact same 2008-9 assignment Is Aplia fit-for-purpose? –What purpose?