1. APPROPRIATELY MEASURING, BENCHMARKING & IMPROVING RETENTION AND GRADUATION IN ODL: Work in Progress at Unisa APPROPRIATELY MEASURING, BENCHMARKING & IMPROVING RETENTION AND GRADUATION IN ODL: Work in Progress at Unisa Herman Visser Senior Specialist: Information & Strategic Analysis Prof George Subotzky Executive Director: Information & Strategic Analysis University of South Africa Presented to NADEOSA Conference, 26 August 2011 1Prof George Subotzky & Herman Visser

2. ACKNOWLEDGEMENTSACKNOWLEDGEMENTS  Original conceptual foundation was done by Herman Visser and refined and expanded based on inputs from Prof George Subotzky  Report ’Improving Throughput at Unisa: A report on three cohort case studies’ by Prof George Subotzky and Herman Visser, August 2008 further refined the conceptual basis  Paper ‘Measuring and Benchmarking Throughput for Improvement’ presented at SAAIR Forum by Herman Visser and Prof George Subotzky, October 2008 also contributed to further conceptual clarification  Analysis of the current cohort throughput and retention per qualification type done by Herman Visser

3. FOCUSFOCUS In this presentation the focus will be on appropriately measuring, benchmarking and improving retention and graduation. The Ministerial targets for distance education will then be tested against the actual retention and graduation patterns in key qualification types at Unisa.

4. CONTEXTCONTEXT  Pursuing more equitable and efficient access, success and throughput has been a prominent goal in South African higher education over the past decades  As dedicated comprehensive distance education institution, Unisa faces particular challenges in this pursuing this goal  The specificities of distance education coupled with past and current life circumstances of students make the task considerably more daunting

5. CONTEXT (Continued)  Key characteristics of distance education students for this analysis:  Higher than average dropouts  Frequent periods of stopouts  Predominantly part-time study  In addition, as in residential institutions, Unisa students are in many cases (if not the majority of cases) inadequately prepared for higher learning

6. MEASURING AND BENCHMARKING THROUGHPUT AND TARGET SETTING  Analysing throughput raises the vexed issue of appropriate measures and benchmarking  This is particularly important in the context of distance education and underprepared students  When we want to improve throughput it is also important to set targets for improvement

7. MEASURING THROUGHPUT: NPHE AND THE PROXY METHOD  National Plan for Higher Education (NPHE) is the primary point of reference for graduation benchmarks in South Africa  Based on specific assumptions and analysis, the NPHE provided an overall framework and specific benchmarks for various qualification types differentiated for contact and distance education  In the absence of detailed cohort studies these benchmarks rely on rough proxy graduation rates and dropout rates  Limitations: enrolment fluctuations; occasional students; active students

8. MEASURING THROUGHPUT: COHORT METHOD  Two main elements by which throughput is measured in cohort studies:  Cohort graduation rate Calculated by the number of graduates over time of a particular cohort of students, normally first time entering students (Unisa also transfer and entering)  Time to completion Relates to the time to complete the qualification  Examples nationally  DHET/CHE cohort studies for all institutions (Scott et al)  HERD-SA project

9. MEASURING THROUGHPUT: COHORT GRADUATION RATE  Graduation rate is directly affected by the course success rate and retention rate  Relationship between course success rate and throughput is complex and needs to be modeled carefully  At Unisa retention is threatened by high dropout and stopout which are both typical characteristics of distance education  Also involves measuring the inverse, namely the dropout/transfer rate, stopout rate and in process rate

10. MEASURING THROUGHPUT: COHORT TIME TO COMPLETION  Measured by the number of years since entering the qualification and the fulfillment of the requirements for the qualification  It is important to take the number of stopout periods into account by excluding them from the time to completion  Time to completion is therefore the elapsed time less the stopout periods  Also involves measuring the inverse, namely the time to dropout/transfer rate, time in process

11. BENCHMARKING THROUGHPUT: BENCHMARKS  In order to interpret the throughput measurements of graduation rates and time to completion meaningfully, appropriate benchmarks must be identified  To ensure their validity, benchmarks needs to realistically take into account the major factors that contribute towards the desired outcomes in a particular context  A benchmark is a reference point appropriate to the context used in the analysis and interpretation of trends, practices and performance

12. BENCHMARKING THROUGHPUT: TARGETS  Important to distinguish clearly between benchmarks and targets as they can be easily conflated  A target is the desired future state, outcome, result or objective of a planned process towards which effort is aimed  Benchmarking is the basis for setting realistic targets  While their may be an aspirational aspect to the setting of a benchmark, its primary purpose is to provide a reference point by which to assess practices and performance

13. BENCHMARKING THROUGHPUT: NATIONAL PLAN ON HIGHER ED. (1)  National Plan on Higher Education, 2001 translated the goals and objectives of the 1996 White Paper on Higher Education Transformation and the 1997 Higher Education Act into a planning framework  This planning framework included identifying benchmark rates for throughput that were differentiated according to qualification type and delivery mode (contact and distance)  Importantly, these benchmarks comprised the proxy method

14. BENCHMARKING THROUGHPUT: NATIONAL PLAN (2)  The calculation of these benchmarks are explained on p 23: [These were calculated] by reviewing student cohort models, involving a combination of retention rates, drop-out rates and graduation rates over a five year period  Supplemented by 2001 Ministerial document Higher Education Graduation Benchmarks in the National Plan for Higher Education - by means of flowcharts and tables, a 5 year analysis of trends set out assumptions for high (NPHE targets), typical and low graduation rate benchmarks

15. BENCHMARKING THROUGHPUT: GRADUATION RATE (3) Three key assumptions regarding distance education:  Final cohort graduation rate for a 3-year UG qualification should be the same as for a similar contact qualification, namely 75%  Final cohort graduation rate for a 4-year UG qualification should be 65% as apposed to 75% because of the extra time involved  Students in 3-year and 4-year distance programmes carried an average course load of 60% of a full-time load each year – this means the minimum time should be 5 years for a 3-year and 7 years for a 4-year distance education qualification

16. BENCHMARKING THROUGHPUT: NATIONAL PLAN ASSUMPTIONS NPHE Distance Education Benchmarks, Graduation and Dropout Rates  Pattern of dropouts for a 3-year UG qualification assumed to be 20%, 19%, 15%, 18% and 22% same as for contact  Assumption that throughput for a 4-year qualification will be lower. However, national and international evidence suggests otherwise.  This strongly suggests that assumptions for contact education was largely applied uncritically to distance education and not based on empirical evidence

17. BENCHMARKING THROUGHPUT: NATIONAL PLAN BENCHMARKS Applying the 90% revision to the NPHE cohort graduation rate benchmarks UG Qualifi- cation Bench- mark NPHE Proxy Grad. Rate NPHE Cohort Grad. Rate NPHE Cohort Dropout Rate Revised Proxy Grad. Rate Revised Cohort Grad. Rate Revised Cohort Dropout Rate 3-yearHigh/ NPHE 15%75%25%13,5%67,5%22,5% Typical/ low 10%35%65%9,0%31,5%58,5% 4 yearHigh/ NPHE 10%60%40%9,00%54,0%36,0%

18. BENCHMARKING THROUGHPUT: MINISTERIAL TARGETS  March 2007 Ministerial enrolment and output targets to be attained by 2010 superseded the NPHE targets with opportunity to institutions to propose changes  Went beyond mere benchmarks and serve as measures of institutional performance – course success rate of 60% and average throughput rate of 10%  After a detailed analysis of trends and realistic potential for improvement Unisa suggested an aggregate throughput rate of 8,37% (which, importantly, excludes occasional students) which was accepted by the Ministry  This target is subject to the limitations of the problematic proxy measures indicated above - increased enrolments, aggregated target don’t account for differentiation by qualification type and level as proposed by Unisa

19. BENCHMARKING THROUGHPUT: CHE NATIONAL BENCHMARKS Institutional Type Graduated within 5 years Still registered after 5 years Left without graduating Universities (excluding Unisa)50%12%38% Unisa14%27%59% All universities38%17%45% Technikons (Excluding TSA)32%10%58% TSA2%12%85% All technikons23%11%66% All institutions30%14%56% The CHE study (Scott et al, 2007) provided the first cohort- based analysis of throughput comparing South Africa’s higher education institutions

20. BENCHMARKING THROUGHPUT: HERD-SA NATIONAL BENCHMARKS 1 st year 2 nd year 3 rd year 4 th year 5 th year 6 th year 7 th year 2000 Baseline enrolment 1 609 Enrolment1 6091 3811 26267231012077 % Dropouts (baseline) 14%7%4% 2% % Cumulative dropouts 14%22%25%29%31%33% % Graduates0%33%19%9%3%0% % Cumulative graduates 0%33%52%61%64% The HERD-SA study (Lourens, 2008) performed a pilot study among a number of South African higher education institutions

21. BENCHMARKING THROUGHPUT: TIME TO COMPLETION  Key factors to take into account for time to completion:  Study load The full-time course load of Unisa students as expressed by FTE Enrolment:Headcount is approximately 0,5 – this does not accord with the NPHE assumption of 60%  Underpreparedness The combined effects of disadvantaged educational background and past and current socio-economic circumstances obstruct success at course level and render students vulnerable to dropout and stopout but is difficult to factor in this aspect  The expected minimum time for each qualification or qualification type are calculated by applying the full time course load

22. BENCHMARKING THROUGHPUT: INTERNATIONAL BENCHMARKS (1)  While the varying contexts and conditions of different regional and national settings preclude making uncritical and inappropriate comparisons, examining a few key international graduation rates benchmarks provides informative reference points  Statistics released by the US Dept of Education (Knapp et al, 2008)  Public 4-year institutions – overall 53,3% cohort graduation rate  Corresponding figures for 2-year public institutions – even lower at 23,2%  In both cases significant differences were reported for the various race/ethnic groupings

23. BENCHMARKING THROUGHPUT: INTERNATIONAL BENCHMARKS (2)  Generally accepted international benchmarks for time to completion is the minimum time plus 50%  However, this does not factor in the two key characteristics of distance education namely part-time study (overall around 50% of a full-time course load at Unisa) and underpreparedness  Tinto (2008) suggests that minimum time plus 50% is an appropriate benchmark to compare the growing number of underprepared students in the United States  Taken together with course load, this suggests an average time to completion of at least double the minimum time for underprepared distance education students

24. CONSTRUCTION OF COHORTS  Normally first-time entering cohorts are constructed in order to isolate only first time entering students  Because of the number of transfers from other institutions and transfers between qualifications in and ODL environment all entering students were considered (First-time entering, Transfer and Entering)  This approach still allow for the creation of a subset of the entering cohort to do analysis in first time entering students but also provides a more comprehensive picture of throughput for a qualification which is a more realistic view, especially for distance education

25. ANALYTIC FRAMEWORK Profile of qualification/qualifications (important for context) Overview of throughput Time to completion Dropout/Transfer out Time to dropout Stopouts (not presented) Time to stopout (not presented) Future component – credits entering students from external transfers and from internal transfers

26. COHORT CONSTRUCTION  Entering cohort  First-time entering students  Transfer students  Entering students  Graduates  Initial qualification  Transfer qualification  Stopouts  Not treated as dropouts

27. First-year dropouts (all qualifications)

28. RESULTSRESULTS The analysis was done for all qualification types but only the main qualification types will be presented Qualification types presented:  Undergraduate Certificates/Diplomas (1-2 years)  National Diplomas  General first Bachelor’s degrees  Professional first Bachelor’s degrees (3 years)  Professional first Bachelor’s degrees (4 years +)  Postgraduate Certificates/Diplomas  Honours degrees

29. FINDINGS: UNDERGRADUATE CERTIFICATES/DIPLOMAS (1-2 YEARS) Minimum timeMostly 1 yr Minimum time plus 50%1,5 yrs Expected minimum time1,6 yrs Graduation rate53,8%-64,6% Overall dropouts oldest cohorts68,9% First-year dropouts of first-time enteringlow 20% - 31,5% Still in process (after 2 yrs)11,7%  Qualification type Undergraduate Certificates/Diplomas was split into two in 2005  Graduation rate above average revised NPHE target  Dropout rate higher than revised NPHE target

30. FINDINGS: NATIONAL DIPLOMAS Minimum time3 yrs Minimum time plus 50%4,5 yrs Expected minimum time5,9 yrs Graduation ratehighest 7% Overall dropouts oldest cohortsin 90% First-year dropouts of first-time entering48,1% - 67,2% Still in process (older cohorts)very small  Graduation rate far below average revised NPHE target  Dropout rate much higher than revised NPHE target  These can be ascribed to the effect of exit-level qualifications up to 2004

31. FINDINGS: GENERAL 1 st B DEGREES Minimum time3 yrs Minimum time plus 50%4,5 yrs Expected minimum time6,4 yrs Graduation rate (after 6 yrs)12% - 16,8% Overall dropouts oldest cohorts73,5% - 73,9% First-year dropouts of first-time entering24,2 % - 33,2% Still in process (after 6 yrs)16,9%  Definition for General First Bachelor’s degrees was changed in 2005 but changed back in 2006  Graduation rate below average revised NPHE target  Dropout rate much higher than revised NPHE target

32. FINDINGS: PROF 1 st B DEGREE (3 years) Minimum time3 yrs Minimum time plus 50%4,5 yrs Expected minimum time5, 7 yrs Graduation rate53,8%-64,6% Overall dropouts oldest cohorts66,7% - 69,4% First-year dropouts of first-time enteringlow 20% - 33,4% Still in process (after 6 yrs)16,1%  Professional First Bachelor’s degrees was split in 2006 to 3-year and 4-year and longer  Graduation rate slightly below average revised NPHE target  Dropout rate higher than revised NPHE target

33. FINDINGS: PROF 1 st B DEGREE (4 years+) Minimum time4 yrs Minimum time plus 50%6 yrs Expected minimum time6,9 yrs Graduation rate15,5% - 19,6% Overall dropouts oldest cohorts48,9% - low 70% First-year dropouts of first-time entering23,7% - 37,1% Still in process (after 7 yrs)15,6%  Professional First Bachelor’s degrees was split in 2006 to 3-year and 4-year and longer  Graduation rate much below average revised NPHE target  Dropout rate higher than revised NPHE target

34. FINDINGS: POSTGRADUATE CERTIFICATES/DIPLOMAS Minimum time1 yr Minimum time plus 50%1,5 yrs Expected minimum time1,8 yrs Graduation rate41,4% - 66,2% Overall dropouts oldest cohortslow 50% - 71% First-year dropouts of first-time entering20,5% - 44,7% Still in process (after 2 yrs) 13% but decline sharply therafter to 4% - 6%

35. FINDINGS: HONOURS DEGREES Minimum time1 yr Minimum time plus 50%1,5 yrs Expected minimum time1,7 yrs Graduation rate37,5% - 63,3% Overall dropouts oldest cohortslow to mid 60% First-year dropouts of first-time entering21,7% - 33,9% Still in process (after 2 yrs) low 10% to 21,7%  Norm not published for distance education  Graduation rate varied – in some cases close to contact target of 65%  Dropout rate higher than revised NPHE target – 45% regarded as unacceptable

36. CONCLUSIONCONCLUSION  Some of the NPHE assumptions needs to be reconsidered based on the empiric evidence  Overall throughput at Unisa needs considerable improvement  Dropouts and stopouts are generally very high  The proportion of underprepared students seems to have increased  The time to completion of students that complete is mostly meeting the target  They main task is to focus on the retention of students and minimising the number of stopouts  A useful framework was created for the measuring and benchmarking of throughput for improvement