1 Standards, quality assurance, best practice and benchmarking in e-learning Professor Paul Bacsich Matic Media Ltd, and Middlesex University, UK

2 The Menu Standards (technical) Quality Assurance –Standards (content) –Standards (pedagogy and process) Best Practice –Excellence? Benchmarking Conclusions

3 Standards (technical) UK follows mainly IMS Agency called CETIS set up by JISC to advise universities and colleges on IMS A few mega universities (OU, Ufi, etc) are direct members of IMS IMS Learning Design gaining influence Also e-portfolios

4 Standards (Content) Quality Assurance Agency has set up “subject benchmarks” More about generalised competences than detailed syllabi See benchmark/

5 Standards (pedagogy and process) Quality Assurance Agency (QAA) “Code of practice for the assurance of academic quality and standards in higher education” See codeOfPractice/ Not much on pedagogy – this is left to the discretion of the professor

6 QAA in e-learning Little has been done specifically on e- learning – but see… “Collaborative provision and flexible and distributed learning (including e-learning)” Recent – September 2004 Some feel it says too little, others do not want to be restricted

7 Digression on Pedagogy Higher Education Academy “works with universities and colleges, discipline groups, individual staff and organisations to help them deliver the best possible learning experience for all students” Runs Subject Centres for each subject Beginning to advise on e-learning

8 Best practice in e-learning Not much studied in the UK yet OU a major source of advice UKeU set up to crystallise best practice into an operational business It failed – but its legacy may help –Committee for Academic Quality US much more active – see e.g. “Quality on the Line” (IHEP, 2000)

9 In UK, universities compete - and now in e-learning Universities want to judge how well they are doing in e-learning And funding agencies also want to know But universities don’t want to tell if they are doing badly! Not the public, not the funding agencies. And universities (like people) are not good at judging themselves.

10 Benchmarking Like Activity Based Costing, it has been around for many years Unlike ABC, but like BPR, quality, excellence, etc; no one is now sure what it means…

11 Back to Basics (Xerox) a process of self-evaluation and self-improvement through the systematic and collaborative comparison of practice [process] and performance [metrics, KPIs] with competitors [or comparators] in order to identify own strengths and weaknesses, and learn how to adapt and improve as conditions change.

12 Benchmarking Dichotomies Implicit Independent Internal Vertical Inputs or Processes Metric Explicit Collaborative [clubs] External Horizontal Outputs Qualitative (After Jackson)

13 Focus of my work Focussed purely on e-learning But not to any particular style (e.g. DL) Oriented to institutions past the “a few projects” stage Suitable for desk research as well as invasive studies Suitable for single- and multi-institution studies

14 Benchmarking (in Universities) There are several reports that will tell you how to do benchmarking in general –Higher Education Academy (UK) –Learning and Skills Development Agency (UK) –Department of Education Training and Youth Affairs (Australia)

15 Benchmarking (in Universities) And some agencies can help: –European Benchmarking Programme on University Management (ESMU, Brussels) –English Universities Benchmarking Club

16 Benchmarking in e-Learning There are very few reports –National Learning Network (UK) –not for universities, but for colleges –E-Learning Maturity Model (NZ) – brand new!

17 Quality/Best Practice in e-Learning There are a few reports (US): –APQC/SHEEO Study 1998 (US) –IHEP “Quality on the Line” 2000 (US) And several projects (EU): –BENVIC –SEEQUEL –Swiss Virtual Lugano: MINE

18 Excellence (?) in e-Learning New project: E-xcellence (EADTU and others) Outside e-learning, several projects: –Consortium for Excellence in Higher Education (UK)

19 Benchmarking e-learning A “synthesis”

20 Processes or Outputs? Outputs first (can be done by desk research) Processes later (best done in clubs or invasive studies) Inputs not of interest to students; but of course of interest to funders

21 Metrics or Bureaucratic Use a 6-point scale –5 from Likert plus 1 more for “excellence” Backed up by metrics where possible Also contextualised by narrative Remember the problems of judging “best practice”; judging “better practice” is easier

22 Other Decisions Explicit (otherwise you are not trying) Independent or collaborative Internal or external Horizontal: focus on processes across the whole institution; do not be seduced into individual projects

23 How Many Benchmarks? It is like ABC: how many activities? Answer: Not 5, not 500. Better answer: Well under 100. –Composite some criteria together –Remove any not specific to e-learning –Be careful about any which are not provably critical success factors.

24 How Many do Others Have? LSDA (UK) has 14 IHEP (US) has 24 APQC/SHEEO (US) had 14 (Breaking news) EMM (NZ) has 43

25 Pick and Mix System 25 criteria (liable to grow to around 30) 6 levels, backed up by qualitative and numeric information Student-oriented Focussed on critical success factors Requires no long training course to understand, if you know about e-learning Methodology-agnostic

26 “Adoption phase” (Rogers) 1.Innovators only 2.Early adopters taking it up 3.Early adopters adopted; early majority taking it up 4.Early majority adopted; late majority taking it up 5.All taken up except laggards, who are now taking it up (or retiring or leaving) 6.First wave embedded, second wave under way (e.g. m-learning after e-learning)

27 “Training” 1.No systematic training for e-learning 2.Some systematic training, e.g. in some projects and departments 3.U-wide training programme but little monitoring of attendance or encouragement to go 4.U-wide training programme, monitored and incentivised 5.All staff trained in VLE use, training appropriate to job type – and retrained when needed 6.Staff increasingly keep themselves up to date in a “just in time, just for me” fashion except in situations of discontinuous change

28 What’s next?

29 Next Steps Correlate with “quality” and “excellence” projects in EU Publish a review report on UK Committee for Academic Quality (in e-Learnng) August Review underpinning methodologies (CMM etc) Literature search outside Europe, US and Commonwealth Series of workshops –at ALT-C 2005 Manchester September –At ACODE Australia November –at Online Educa Berlin December

30 Thank you for listening Any questions? Professor Paul Bacsich Global Campus, Middlesex University