1. Degree completion as an indication of quality Nigel Palmer College of Arts & Social Sciences Australian National University International Conference on Developments in Doctoral Education & Training UK Council for Graduate Education Examination Schools, University of Oxford Monday March 30 2015

2. 2 Overview This paper responds to the question: Can degree completions be an indication of quality within and between educational systems? …and concludes that: Degree completions information may be a better indicator at the institutional level for process rather than outcomes when it comes to quality.

3. 3 Overview This paper responds to the question: Can degree completions be an indication of quality? …using the following headings: 1.Degree completions as a policy metric; 2.Degree completions as a valid metric; 3.Degree completions as a reliable metric; 4.Some concluding thoughts on quality…

4. 4 Acknowledgements This paper draws on findings from two benchmarking exercises: The first using the Gap Analysis template developed for the Good Practice Framework and conducted as part of an Office of Learning and Teaching extension project supported by the Australian Government, and The second conducted for The Australian National University. This paper arises from a doctoral project entitled Benchmarking Graduate Study (see QPR 2016). Acknowledgments and thanks go to Joe Luca, Inger Mewburn and Margaret Kiley respectively for supporting each of these initiatives.

5. Overview This paper responds to the question: Can degree completions be an indication of quality? …using the following headings: 1.Degree completions as a policy metric; 2.Degree completions as a valid metric; 3.Degree completions as a reliable metric; 4.Some concluding thoughts on quality… 5

6. 6 Indicators used in the evaluation of doctoral programs in Europe (findings from the EUA Accountable Research Environments for Doctoral Education study) Source: Byrne, Jorgensen & Loukkola (2013) Degree completions as a policy metric

7. 7 Criteria in determining Australian Government support for research education through the Research Training Scheme (RTS) Source: Palmer (2013) Degree completions as a policy metric 1988-2001 Block Grants % 1999 RTS (as proposed) % 2002-2015 RTS (as implemented) % Publications 4.6 / 4.0 - 10 Research income 35.4 / 32.03540 RHD completions 203550 Existing places / load 4030- While no longer used in determining RTS grant amounts research student load has subsequently been used in determining block grant allocations for related programs (DIISRTE, 2012). See also ‘smoothing effect’.DIISRTE, 2012

8. Overview This paper responds to the question: Can degree completions be an indication of quality? …using the following headings: 1.Degree completions as a policy metric; 2.Degree completions as a valid metric; 3.Degree completions as a reliable metric; 4.Some concluding thoughts on quality… 8

9. 9 Ratio of total PhD to bachelor degree completions in Australia 2001-2009 (top 8 as of 2009) Source: Selected Higher Education Statistics (various years) (Department of Education, 2013).Department of Education, 2013 Degree completions as a valid metric

10. Overview This paper responds to the question: Can degree completions be an indication of quality? …using the following headings: 1.Degree completions as a policy metric; 2.Degree completions as a valid metric; 3.Degree completions as a reliable metric; 4.Some concluding thoughts on quality… 10

11. 11 Can degree completions provide reliable information? A few methods for calculating completion rates compared: –Raw completions (total numbers); –Ratio approaches (relative to commencements); –‘Crude’ measures (relative to cohort); –Commencement-driven cohort approaches. Degree completions as a reliable metric

12. 12 Criteria in determining Australian Government support for research education through the Research Training Scheme (RTS) Source: Palmer (2013) Degree completions as a reliable metric 1988-2001 Block Grants % 1999 RTS (as proposed) % 2002-2015 RTS (as implemented) % Publications 4.6 / 4.0 - 10 Research income 35.4 / 32.03540 RHD completions 203550 Existing places / load 4030- While no longer used in determining RTS grant amounts research student load has subsequently been used in determining block grant allocations for related programs (DIISRTE, 2012). See also ‘smoothing effect’.DIISRTE, 2012

13. 13 Comparison of completion ratio measures for Australian universities Source: Selected Higher Education Statistics (Students) for the relevant year, Department of Industry, Innovation, Science, Research and Tertiary Education. Ratios are expressed as percentages. Only university providers reporting data since 2001 were included for comparison. Degree completions as a reliable metric

14. 14 ‘Crude’ research doctoral degree completion rates 2002-2011 (top 8 for period)* Source: Selected Higher Education Statistics (various years) (Department of Education, 2013) *Excludes providers with under 200 doctoral enrolments.Department of Education, 2013 Degree completions as a reliable metric

15. 15 Findings from the US PhD Completion Project: Cumulative Completion by Broad Field of Study Source: Sowell (2008) Degree completions as a reliable metric

16. 16 HEFCE studies: Time to PhD completion or last PhD activity for ‘full-time’ Research Council students who began their studies in 1996-97 Source: HEFCE (2007) Degree completions as a reliable metric

17. 17 Single institution study: ANU Completion rates for ‘full-time’ PhD candidates by scholarship status (commencing cohorts 1985-2007) Source: Rowley and Nurbasari (2011) Degree completions as a reliable metric

18. 18 Single institution study: Cumulative program status at year 9 for ANU research doctoral candidates commencing in 2004* *Cumulative program status as of July 2013. Source: Australian National University unpublished internal data. Degree completions as a reliable metric

19. 19 Source: Australian National University unpublished data. Single institution study: Cumulative Completion rates for 2004 doctoral commencements for ANU international and domestic candidates by gender Degree completions as a reliable metric

20. 20 Can degree completions provide reliable information? There appears to be good alignment between commencement- based cohort approaches to calculating completion rates, both in terms of findings and methods; Does this mean we can be more confident in using this approach in comparing findings within and between educational systems? A couple of challenges still remain… –Using commencing status to distinguish ‘full time’ and ‘part time’ comparison groups –Hall, Evans and Nerad (2006) –Accounting for the ‘completions process’ Degree completions as a reliable metric

21. 21 Can we reliably benchmark completion rates or times within and between educational systems? Degree completions as a reliable metric

22. 22 Process benchmarking using the Gap Analysis Template developed as part of the Good Practice Framework. The project involved the following benchmarking partners: Edith Cowan University Curtin University Victoria University University of Tasmania James Cook University Degree completions as a reliable metric

23. 23 Steps: Develop a ‘laundry list’ of specification for a custom dataset, including a range of potential ‘completion events’; Liaise with specialist data units within each of the participating institutions; Compare trends, alignment and utility of the data elements collected and the indicators that may be derived from them. Degree completions as a reliable metric

24. 24 Research degree ‘completion’ can refer to: 1.Submitting a thesis for examination; 2.The final date of enrolment; 3.Receipt of final thesis examination reports; 4.Satisfaction of all requirements for conferral of a degree; and 5.Degree recorded as conferred by the education provider.

25. Six enrolment events associated with a completions process: Degree completions as a reliable metric

26. 26 Data elements benchmarked for evaluating the completions process (16 completion + 3 status indicators) Degree completions as a reliable metric

27. 27 Findings from process benchmarking: Preliminary findings show the enrolment events outlined in the previous table more or less follow in sequence; Most events incrementally contribute to the overall completion time; Comparable mean times between similar enrolment events were found in some (but not all) cases across institutions; An acceptable level of comparability in data recording practices across institutions needs to be established before findings of this kind can be used for internal benchmarking purposes, and this may be some way off... Degree completions as a reliable metric

28. 28 What would a staged analysis of completion events look like? (Constructed using sample data unrelated to the benchmarking project) Degree completions as a reliable metric

29. Overview This paper responds to the question: Can degree completions be an indication of quality? …using the following headings: 1.Degree completions as a policy metric; 2.Degree completions as a valid metric; 3.Degree completions as a reliable metric; 4.Some concluding thoughts on quality… 29

30. 30 Some concluding thoughts on quality… Quality: standards, outcomes or fitness for purpose? The use of metrics: A horse without a rider? Conclusion: Does information on completions offer an indication of quality?

31. Degree completion as an indication of quality Nigel Palmer College of Arts & Social Sciences Australian National University International Conference on Developments in Doctoral Education & Training UK Council for Graduate Education Examination Schools, University of Oxford Monday March 30 2015