1. © University of Reading 2012 www.reading.ac.uk Statistical monitoring of student performance – an early warning system Dr Karen Ayres, Dr Nick Biggs & Dr Paul Glaister Department of Mathematics and Statistics University of Reading

2. 2 A need for monitoring students Performance in the first year of a degree influences performance later on. Some students struggle with the demands of university study. Reticence, apathy or embarrassment may mean they do not seek help. Either they drop out or they may not go on to achieve their full potential.

3. 3 An early warning system is needed! Useful tools for monitoring progress are formative or summative problem sheet marks. Sudden drops or a prolonged decline in marks may indicate a problem. Making use of such data, possibly in a visual way, can help flag potential problems as they arise. Profiles of marks can also become a talking point in tutor meetings.

4. 4 Mark profiles... for a whole cohort In our first year, we have about 140 students taking different subsets of up to 7 maths and stats modules in a single 10-week term. For just 5 students and one module... To flag clearly which are potential issues, we propose using control chart methodology to highlight profiles...

5. 5 Control charts Control charts are widely used in statistical process monitoring, to see if the process is ‘out of control’ (e.g. a shift in the mean). The situation is different from ours, in that usually an entire run is complete before the chart is produced. We propose creating a chart each week. The charts are there to flag possible issues - an aid to tutors in monitoring students.

6. 6 P-charts for monitoring performance A p-chart plots a sequence of proportions p i... for us it is the mark x i out of 10 for week i. For any week we have the average p over all current and previous weeks where n i = n = 10. Also where k = 2 or 3.

7. 7 P-charts for monitoring performance, ii A warning flag is raised if p i is lower than LCB.

8. 8 Individual Moving Range charts... We focus only on the individual chart in an Individual Moving Range chart approach. For normalised marks x i, calculate (over m weeks) where k = 2 or 3, d 2 = 1.128, and A warning flag is raised if x i is lower than LCB.

9. 9 Which is best? We simulated 200 profiles under various scenarios, including – 1a. Student mark randomly varies about 40% – 1b. Student mark randomly varies about 80% – 2. Student mark randomly varies about 80% except for two difficult weeks for whole class (about 40%) – 3. Slow decline over weeks (varying from 80% to 40%) – 4. Sudden drop at week 8 (varying about 80%, to 40%)

10. 10 Percentage of profiles flagging an issue Sim. P-chartImRBoth P and ImR False positives (k = 2) 1a. 18%29.5%10.5% 1b. 41%47.5%33.5% 2. 100%40%11.5% True positives (k = 2) 3. 95.5% 92%89% 4. 92.5% 89%86% Preferred

11. 11 A practical implementation... Our aim is to use the charts (and the ideas behind them) to flag potential issues – for use by the Senior Tutor – for use by the personal tutor The problem? Recall... we have ~140 students and 7 modules in one term with weekly problem sheets in our first year!

12. 12 A practical implementation, ii In Excel – All data are entered by the School Office admin staff – For each module, cells are highlighted automatically when the two bounds are exceeded for a student – Graphs are produced of the profiles per module for all of a tutor’s tutees, or for all modules per tutee – At the click of a button, the marks profile is emailed to each personal tutor (with or without graphs) – At the click of a button, all flagged profiles are emailed to the Senior Tutor

13. 13 A demonstration Our focus so far has not been to produce the actual control chart, but to use the method to flag issues en masse. Let’s see the Excel program in action...

14. 14 E-mails Simple email to the tutor (mark profile only) Email to the tutor with attachments Email to the Senior Tutor with flagged profiles

15. 15 Future directions We will trial this in 2012-13 with our students. It will be important to collect complete and accurate data! We may add facility to produce control charts with limits for a given student and module. Future work could focus on multivariate extensions – can consider multiple modules at once for evidence of consistent drop in performance.

16. 16 Thank you for listening Any questions? Excel file free to download from http://www.personal.reading.ac.uk/~sns99kla/