1. From evaluation to reflection-on-action: Lessons learnt from the impact of a distance education programme NADEOSA Conference (24 – 25 June 2014) Ruth Aluko

2. Introduction Quality assurance at various levels in HE, but a gap  evaluation dimension (Mizikaci, 2010) Evaluation in HE (Hall & Hall, 2004)  focus – aims of a program & to what extent  tool for decision-making & assessing quality Purpose  to maintain / improve quality of products & processes Tool  reflection 2

3. Background ACE (EM) - Continuing professional qualification Paper-based DE program Elongated impact study (2007 – 2012)  Mixed-methods enquiry (surveys & interviews)  Surveys: 300 graduates (2004-2006 cohorts) 3

4. Background 128 principals  Interviews (2 individual & 10 focus group) 30 teachers 20 principals  Kirkpatrick’s & Baldwin & Ford’s Training Evaluation Models  Evidence of value added to individual lives & workplace  Suggestions were proffered 4

5. Reflection & Reflective Practice By three methods we may learn wisdom: first, by reflection, which is noblest; second, by imitation, which is easiest; and third, by experience, which is the bitterest. (Confucius) Definition: “active, persistent and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it and the further conclusions to which it tends” (Dewey, 1933) Reflection-in & on-action (Schön 2011; Hall & Hall, 2004) Trigger for reflection (Pammer, et al., 2012) 5

6. Cyclic Process of Reflection ContentReturn Feelings & Emotion Re- evaluation Possible explanations Outcome Intelligent action 6

7. Reflection Questions What valuable lessons can be learnt from the impact of the discontinued ACE: (EM) programme on the professional development of graduates and why? How could these lessons apply to other distance education programmes at the University of Pretoria, and elsewhere? 7

8. 6 Major Lessons 1.Clear institutional policy on quality assurance  persistent negative perceptions of DE programmes (within & outside)  Quality – a priority & an iterative process  Awareness among stakeholders  Possible relationship between student attrition rates & quality 8

9. Table 1:Non-completion rates of distance education students at UP (All programmes) Year Students who discontinued their studies Percentage of total enrolment 2006 4194% 2007 6504% 2008 5843% 2009 7624% 2010 4302% 2011 6503% 2012 6894% 2013 3362% Source: University of Pretoria (UDE) (2006–2013) 9

10. 2.Practices guided by a policy on distance education  Traditional Education policy does not necessarily apply to DE  Importance of policy document 3. On-going monitoring of the DE students’ profile  Necessity for quality data to make data-driven decisions regarding programme improvement 10

11. Examples of data: age, technology, work & geographical profiles, and graduation & retention rates. 11 Table 2: Technology profile of students who enrolled for the first time between 2004 and 2013 Year2004200620072008200920102012 2013 Number of students3 1875 0875 6438 0116 1025 6753 3542 599 Cell phone use98%99% 100% Internet use0%2%1%3%8%13%25%29% Source: University of Pretoria (between 2004 & 2013)

12. 4. Student support structures  all activities beyond the production and delivery of course materials  Most students from TE background  Examples: contact sessions, tutorial letters, assignments, SMS & an academic enquiry service 5. Programme design  Continual review  In spite of IT, most countries in Africa are still trapped in the first-generation mode of delivery 12

13. 6. Research focused on DE  Dedicated research unit  Operational & academic research  Purpose is to inform practice  understanding trends and issues in terms of topics & methods E.g. mixed-methods 13

14. Concluding Thoughts Quality and quality management are organisational obligations that lie vested in management’s commitment towards an understanding of quality (Aluko, Fraser, & Hendrikz 2008) improved performance of distance education students attested to by scholars if quality improves The necessity for continual evaluation of DE programs & reflection-on-action 14

15. Bibliography Aluko, R. 2009. The impact of an Advanced Certificate in Education (ACE) programme on the professional practice of graduates. The International Review of Research in Open and Distance Learning, IRRODL, 10 (4) www.irrodl.orgwww.irrodl.org Aluko, FR, WJ Fraser and J Hendrikz. 2008. Some interfaces in conventional and distance education programmes in a postmodern context. South African Journal of Higher Education, 22(3): 484–497. Dewey, J. 1933. How we think: A restatement of the relation of reflective thinking to the educative process. Boston: D.C. Heath. Hall I. and D. Hall. 2004. Evaluation and social research: introducing small-scale practice. United Kingdom: Palgrave Macmillan. Lee, Y, MP Driscoll and DW Nelson. 2004. The past, present, and future of research in distance education: Results of a content analysis. American Journal of Distance Education, 18(4): 225– 241. Mizikaci, F. 2010. Total quality management in higher education: An evaluation model for practitioners. Germany: LAP Lambert Academic Publishing. Pammer V., K. Knipfer, B. Krogstie, D. Wessel, M. Prilla and S. Lindstaedt. Reflective learning at work – a position and discussion paper. 2012. Available from http://ceur-ws.org/Vol- 790/paper4.pdf.http://ceur-ws.org/Vol- 790/paper4.pdf Rubin, F. 1995. A basic guide to evaluation for development workers. Oxford: Oxfam Publications. Schön, D.A. 2011. The reflective practitioner: how professionals think in action. England: Ashgate Publishing Limited. Wilson-Strydom, M. 2004. Programme evaluation and its role in quality assurance. Vancouver: Commonwealth of Learning. Available at: http://dspace.col.org/handle/123456789/351http://dspace.col.org/handle/123456789/351 15

16. Thank You! ruth.aluko@up.ac.za ruth.aluko@up.ac.za