28-Nov-18 Benchmarking e-learning in UK Universities: Reflections on University of Leicester’s participation in the HEA-led benchmarking of e-learning (2005-06) Jaideep Mukherjee Beyond Distance Research Alliance University of Leicester Presentation to the ALT-C 2006, Herriot Watt University, Edinburgh, 5 Sep 2006 28-Nov-18

The HEA’s rationale for the project: 28-Nov-18 The HEA’s rationale for the project: Through this exercise, the HEA are attempting to understand ‘more about institution’s own, and the sector's, progress in e-learning’; The  exercise had three stated aims: ‘to provide institutions with information to make informed plans for future development’, ‘to allow institutions providing HE to identify their current progress, on embedding e-learning, in relation to similar institutions’, ‘to provide a picture of the sector as a whole in order to identify areas of strategic importance to inform the work of the JISC, the HEA and the Funding Councils’. The exercise was expected to yield quantifiable and qualitative information on how far HEI-s were 'embedding' aspects of e-learning into institutional policy and practice. 28-Nov-18

Univ. of Leicester’s rationale for participation: 28-Nov-18 Univ. of Leicester’s rationale for participation: UoL is an established player in distance learning and e-learning projects; Had already made a strategic commitment to e-learning, which included: Substantial internal consultation and self-reflection; Scan of competitor activity; and Massive increase in VLE use. A senior member of staff at UoL had led the HEFCE benchmarking programme; Had developed and adopted (in 2004-2005) an ‘e-learning strategy’, fully embedded in UoL’s ‘Teaching and learning strategy’ and ‘Distance Learning strategy’ documents; and UoL was keen to assess its position in e-learning among comparable providers (i.e. comparators) 28-Nov-18

The HEA benchmarking e-learning project: 28-Nov-18 The HEA benchmarking e-learning project: Benchmarking project and Pathfinder Programme announced by HEA at ALT-C (5-8 Sept 2005) Expressions of Interest call (Oct – Nov 2005), HEA noted that 56 responses were received; Selection process for Consultants (Oct – Dec 2005), HEA selected 3 Consultants from 23 bids received; 12 pilot institutions selected and notified in Dec 2005; Launch meeting for pilot institutions (20 Jan 2006) University of Leicester (UoL) allocated to Prof. Paul Bachsich’s benchmarking ‘club’, along with the Universities of Chester, Manchester and Staffordshire; Reported back to HEA on (21 Jun 2006) with findings from the Benchmarking exercise, along with other institutions. 28-Nov-18

UoL: Expected outcomes 28-Nov-18 UoL: Expected outcomes Research into e-learning; Measures of equivalence of experience for distance learners and campus-based students; Rewarding and celebration of staff involved in teaching and learning innovation and transformation associated with ICT; Use of the VLE-s and post-VLE tools across all departments. Planning system that considers all relevant market, pedagogies and technology options for new programmes; 28-Nov-18

UoL: Expected outcomes (cont’d) 28-Nov-18 UoL: Expected outcomes (cont’d) Level of knowledge throughout the University of e-learning issues in line with the governance model of a research-led university; Regularly updating e-learning and distance learning costed business plans; Regularly updating market research and competitor analysis reports; Measuring the students’ satisfaction with e-learning; Rating ourselves against the HEFCE ‘Measures of Success’, and Other measures proposed by expert commentators on benchmarking e-learning 28-Nov-18

The actual exercise: Engaging the HEI 28-Nov-18 The actual exercise: Engaging the HEI High-level engagement – e.g. PVC (L&T), key admin directors, service directors; At the coalface – e.g. e-learning experts, DL departments in transition, departmental heads; Structured meetings – via steering groups, individual consultations and access; Scope and time for negotiating on criteria and scoring; Reporting to the HEA– blogs, not reports; Sharing information with ‘club’ members and other HEI-s; Interacting with the consultant Using internal and external (consultant) feedback 28-Nov-18

Methodology: The ‘Pick and Mix’ approach 28-Nov-18 Methodology: The ‘Pick and Mix’ approach According to Prof. Bacsich, this approach: Is focussed purely on e-learning, but could be extended more widely; Draws on several sources and methodologies; Is not linked to any particular style of e-learning (e.g. distance or on-campus or blended); Is oriented to institutions past “a few projects”; Is suitable for desk research as well as “in-depth”; Is suitable for single- and multi-institution studies; N.B. The initial version - developed in early 2005 – has since been strengthened – ‘by literature search, discussion, feedback, presentations’ (See: http://www.heacademy.ac.uk/weblogs/benchmarking/?p=99 ) 28-Nov-18

Examples of (core) criteria and scoring: 28-Nov-18 Examples of (core) criteria and scoring: 28-Nov-18

Examples of (supplementary) criteria and scoring: 28-Nov-18 Examples of (supplementary) criteria and scoring: 28-Nov-18

Typical instruments for collecting data: 28-Nov-18 Typical instruments for collecting data: 28-Nov-18

Reporting instruments: 28-Nov-18 Reporting instruments: HEA blogs - http://elearning.heacademy.ac.uk/weblogs/benchmarking/ Gathering and analysing evidence: 28-Nov-18

Benchmarking institutional ‘slices’: 28-Nov-18 Benchmarking institutional ‘slices’: Why benchmark departmental ‘slices’? Early decisions on which departmental ‘slices’ complement the ‘institutional’ benchmarking; At UoL ‘slices’ include: School of Biological Sciences (including Genetics CETL) Management Centre Department of Media and Communication Research Leicester Medical School Key differences when benchmarking against same criteria 28-Nov-18

Recording, analysing and scoring responses: 28-Nov-18 Recording, analysing and scoring responses: Stages of the data collection and analysis process: Briefing at Steering Group meeting, including consultant Negotiating and agreeing criteria and matrices Background information collection and review Interview(s) with key institutional or departmental nominee(s) Further data / document collection Analysis of interview and document data Cross-referencing information and developing the narrative Tentative score allocation Rehearsal meeting Steering Group meeting with consultant to finalise scoring Reporting on scores and process to the HEA and other institutions involved 28-Nov-18

Operational and methodological issues: Some observations 28-Nov-18 Operational and methodological issues: Some observations High-level institutional buy-in essential; Clear institutional aims and objectives required early in the benchmarking process; Issues of institutional confidentiality; Issues of interpreting criteria and scoring; Avoiding ‘self-congratulatory’ responses; Means of intra-institutional feedback. 28-Nov-18

Operational and methodological issues: Some observations (cont’d) 28-Nov-18 Operational and methodological issues: Some observations (cont’d) Means of inter-institutional information sharing… or lack of it thereof … little opportunity yet of learning from excellence in this sector; Addressing student experiences through benchmarking e-learning ‘We don’t do central!’ – One methodology does not fit all UK HEI-s; Means of addressing aspects of e-learning where UoL ‘fell short’ on scores, and UoL’s Pathfinder proposal. 28-Nov-18

28-Nov-18 Postscript: HEA have now invited another 35 UK HEI-s to undertake the benchmarking exercise in their institutions; The same set of (5) approaches are available for these institutions to use; The 12 original benchmarking institutions have now embarked on the ‘pathfinder’ phase of this project. The University of Leicester’s ‘pathfinder’ project is titled ADELIE: Advanced Design for E-Learning - Institutional Embedding. The aim of this project is to build institutional capacity in the areas of designing for e-, m- and blended learning and teaching online with and through student activities 28-Nov-18

Thank you for listening. Questions and comments welcome. 28-Nov-18 Thank you for listening. Questions and comments welcome. For further queries/comments: Jaideep Mukherjee jm118@le.ac.uk Please visit us at: www.le.ac.uk/beyonddistance 28-Nov-18