1. © The Open University, Institute of Educational Technology 1 Alison Ashby, Naomi Jeffery, Anne Slee Student Statistics and Survey Team, IET The Open University What Works? Student Retention and Success Two Day Retention Convention 3 rd and 4 th March 2010 Using Institutional information to understand student retention and support enhancement

2. © The Open University, Institute of Educational Technology 2 Background information The Open University (OU) is the UKs largest university and teaches 35% of all part- time undergraduates in the UK. There are over 150,000 student course registrations each year (over 75,000 FTE). The OU has a diverse student population, for example: just over 40% of students have less than two A levels or equivalent on entry (the OU has an open entry policy) Just over 10% of students are aged 55 or over and 14% are aged under 25. There are 450 course/modules at undergraduate level Source Facts and Figures 2007/08 The Open University

3. © The Open University, Institute of Educational Technology 3 The case for a course pass model Comparisons of course pass rates take no account of the scale and diversity of students and the curriculum. It is easy to explain away differences in course pass rates by suggesting they are due to different course types/students. Attempts to explore differences in a univariate way are too complex and too resource intensive given the scale. A course pass rate model: –controls for known factors about students and courses –provides a fairer comparison of course pass rates –enables the University to focus attention on courses that have higher/lower pass rates than expected. –Increases understanding / opportunities for enhancement

4. © The Open University, Institute of Educational Technology 4 Key characteristics of the model The model predicts the probability of an individual student passing a course. Separate models produced for new and continuing students. For each course, individual probabilities are added together to provide a predicted pass rate for a course. Each variable included in the model is treated as a categorical variable e.g. age is converted to age bands. Logistic regression has been used to produce the model. The current model is based on five years of data. A statistic called the Z score, which takes into account course population, is produced to compare actual pass rates with predicted pass rates. This is used to determine which courses appear to be under/over performing.

5. © The Open University, Institute of Educational Technology 5 Examples : student characteristics gender age occupation previous study ethnicity qualifications disability financial support

6. © The Open University, Institute of Educational Technology 6 Examples: Course characteristics Course in first year of presentation credit points Number of assignments level Exam / end of course assessment Academic unit

7. © The Open University, Institute of Educational Technology 7 Model benefits and limitations Benefits: More meaningful comparison of course pass rates Systematic process for identifying courses where there may be opportunities for enhancement or sharing good practice. Limitations: Models are approximations Some important factors may be missing from the model The model, on its own, could excuse performance based on known factors. e.g. the model predicts a lower course pass rate where a higher percentage of students have characteristics which have a negative effect

8. © The Open University, Institute of Educational Technology 8 An example of a predictor variable (scale = maximum likelihood estimates, blue bar = standard errors)

9. © The Open University, Institute of Educational Technology 9 How is the course pass rate model used at the Open University? At a local level by course teams as part of the annual course review process. At University level, by the Student Experience Advisory Group (SEAG) which has responsibility to advise on : The effectiveness of the use of information on the student experience for quality enhancement purposes. Areas of particular strength and weakness in the quality of the student experience and performance. The results of the model are not used in isolation. Given the limitations, it is important to triangulate results.

10. © The Open University, Institute of Educational Technology 10 Annual Course Review In addition to the results of the course pass rates model, the following data sources are available to course teams: –Actual registrations and course pass rates over a five year period for all courses to allow a review of trends and comparisons with other courses. –A profile for each course which includes the distribution and pass rates analysed by key indicators e.g. age, gender, previous courses taken and credit points gained. –A table showing results for the ten key performance indicators identified from the end of course survey for each course surveyed in the period under review. –Detailed survey results for each course –Open comments for each course

11. © The Open University, Institute of Educational Technology 11 Addressing the evidence Briefing sessions take place to support course teams in using and interpreting the information provided. As part of the review process course teams are asked to consider the evidence sources and comment on: The strength of the course and actions planned/being taken to share successful practice of the course in its programme of study/academic unit areas for improving the course which have been identified and actions planned/being taken to address them Each academic unit has responsibility for ensuring the oversight of these reviews and the actions planned/taken.

12. © The Open University, Institute of Educational Technology 12 The Student Experience Advisory Scrutiny Group (SEAG) SEAG has agreed a set of three key performance indicators to select annual course review reports for scrutiny: The course pass rate model : Z score outside acceptable value thresholds Absolute pass rate of one standard deviation above or below the mean for level 1, level 2 and level 3 courses. Overall satisfaction/dissatisfaction with the quality of the course is equal to or above pre-determined thresholds (for the last presentation of the course surveyed) A detailed report is prepared for SEAG and a summary for the Learning and Teaching and Student Support Committee

13. © The Open University, Institute of Educational Technology 13 Some examples: via central academic units/ SEAG Good practice FAQs on student course forums Use of wikis, teleconferences and forum notes to support associate lecturers Extensive use of student and tutor online discussion which led to improved ICT skills among students Areas for improvement Student workload higher compared to similar courses Blended learning less well received compared to other new courses

14. © The Open University, Institute of Educational Technology 14 An example of action taken at course level A third level course team responded to student feedback on workload by: Making some material optional Changing the assessment strategy Improving the guidance on assessment Results Pass rate increased by 10% points and the Z score, although still negative, improved substantially in 2008. Additional changes in 2009:further advice on study skills and exam added to course website. Slight improvement in Z score but no change in pass rate.

15. © The Open University, Institute of Educational Technology 15 To summarise : Closing the feedback loop? Data tools Data tables Open comments Key Performance Indicators Academic reviews VCE / University Committees Strategic projects External audits Actions to improve Sharing good practice Collecting/measuring Student Retention Student Feedback

16. © The Open University, Institute of Educational Technology 16 Next steps Continue to increase the focus on identifying and monitoring action where improvement required. Continue to improve the identification and sharing of good practice through cross unit staff development. Continue to develop the evidence base e.g. use of data from the Virtual learning Environment (VLE). Explore a theme based approach to enhancement. Develop a model for student progression.

17. © The Open University, Institute of Educational Technology 17 The good news! Completion rates have increased in 2008/09 compared to 2007/08. The University has received consistently high ratings in the National Student Survey, 94% of respondents were satisfied in 2009. There are a number of strategic projects aimed at enhancing the student experience and increasing retention and progression.

18. Institute of Educational Technology The Open University Walton Hall Milton Keynes MK7 6AA www.open.ac.uk/iet