Automated feedback in statistics education Faculty of Science Freudenthal Institute Automated feedback in statistics education Sietske Tacoma Summer School Freudenthal Institute August 16, 2017

Challenges in university statistics education Statistics is difficult Many abstract concepts Relations between concepts Statistics anxiety Groups are large Statistics relevant for almost all areas of science Difficult to provide individual guidance and feedback Castro Sotos, A. E., Vanhoof, S., Van den Noortgate, W., & Onghena, P. (2007). Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education. Educational Research Review, 2(2), 98-113.

Solution direction Solve statistics problems in digital learning environment Digital Mathematics Environment (DME) Simulations, interactive graphs, contexts (Much more on the DME tomorrow morning) Provide automated feedback Local: feedback on intermediate steps Global: feedback on series of tasks

Project outline Educational setting Introductory statistics courses as Utrecht University Biology (230), Economics (450), Social Sciences (1200) Phase 1 – design based research Design of two feedback types (local and global) Pilots: evaluate feedback and student experiences Phase 2 – evaluation study Influence feedback on student knowledge and understanding Interplay between two feedback types

This talk: local feedback in hypothesis testing Introduction/recap hypothesis testing Stepwise solutions Designing feedback for intermediate steps Freudenthal Instituut | Automatische feedback in universitair statistiekonderwijs

Introduction/recap hypothesis testing How would you react if the grade you received for an exam is much lower than you had expected? Research suggests that most students think they can handle such situations better than their peers, but some students think their coping is worse than that of their peers. In this study, participants were asked to read a scenario of a negative event and indicate how this event would influence their well-being (-5: worsen much, +5: improve much). Next, they were asked to imagine this same event from the perspective of a peer. The difference between both judgements was noted. Suppose that for the sample of n=25 students the mean difference score was MD=1.28 points (own judgement minus judgement peer) with standard deviation SD=1.50. Based on these data, can you conclude that there is a significant difference between the own judgements and judgements of peers? Use a test with α=.05. Freudenthal Instituut | Automatische feedback in universitair statistiekonderwijs

Difficulties in hypothesis testing Reasoning under assumption that null hypothesis is true Dealing with variability sample mean Relation between values (test statistic) and probabilities (in probability distribution) Freudenthal Instituut | Automatische feedback in universitair statistiekonderwijs

Stepwise solutions in the DME Freudenthal Instituut | Automatische feedback in universitair statistiekonderwijs

Stepwise solutions - advantages Feedback on every step Possibility to address common errors Structure for students VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221.

Stepwise solutions - disadvantages Structure for students Focus on procedural skills

Design: freedom to choose steps

Designing feedback for intermediate steps Feedback on separate steps “incorrect alternative hypothesis” Address common errors Feedback on order of steps “to draw a conclusion, you first need to specify a critical value” Structure constructed by students themselves Mechanism: domain reasoner

Domain reasoners for stepwise feedback Buggy rules: error recognition Feedback Partial solution & previous partial solution no Constraints: correct solution path? Rules: solution path recognition Feedback yes Hint Heeren, B., & Jeuring, J. (2010). Adapting mathematical domain reasoners. In S. Autexier, J. Calmet, D. Delahaye, P. D. F. Ion, L. Rideau, R. Rioboo & A. P. Sexton (Eds.), Intelligent computer mathematics (pp. 315-330) Springer Berlin Heidelberg.

Demo DME test environment

Designing feedback text What feedback would you give?

Designing feedback text What feedback would you give? Try to stimulate thinking and reasoning Address concepts Not give away correct answers

Further plans September – november 2017: 200 students 6 tasks Evaluate whether feedback and hints are appropriate Evaluate whether feedback and hints help 2018: Use by 1500 students (social sciences and economics) Inform student model and adaptation