JISC LLL WFD Assembly Maximising Impact – assembling & leveraging the evidence base Dr Rachel Harris Inspire Research Ltd 10 th January 2011

Inspire Research Ltd Curriculum Delivery & Evaluation The programme invited projects to “transform how they deliver and support learning across a curriculum area through the effective use of technology”. 13 projects funded by JISC, and 2 by Becta –What did they want to know? –How did they do it? –What impact did they have? (And, what was the evidence?) Project outputs: Project blogs aggregated at:

Inspire Research Ltd Delivery evaluation map Live map

Inspire Research Ltd Impact on the bottom line eBiolabs – no more breakages or how to balance a centrifuge Cascade – reduced admin time & the evidence to prove it eReflect – students engage with assessment feedback

Inspire Research Ltd eBiolabs Since introduction of eBiolabs, student lab prep time increased from 20 to 35 minutes (average). 54% students agreed they “felt well prepared when they entered the lab” compared to 18% prior to introduction of eBiolabs. Designed to support one 40 credit unit in one School. Now supports ten units across the Faculty of Medical and Veterinary Sciences. Approx 120 teaching staff impacted by use of eBiolabs at Bristol. Currently, 900 student users.

Inspire Research Ltd Cascade – Oxford University “We forecast savings equivalent to over 38 weeks of productive admin time a year.” (Savings identified via historical data and time motion studies.) “These efficiency savings, and our ability to quantify them, have been useful when we have needed to make the case to fully embed and sustain these services going forward.”

Inspire Research Ltd eReflect – Westminster University

Inspire Research Ltd “The e-Reflect feedback suggested that I speak to the module leader … I wouldn’t normally have thought to do this and it helped. He gave me ways I could improve and suggested a different way of approaching some of the subject matter. It has helped I think because since then my marks have shot up.” Link to video Link to video

Inspire Research Ltd Evaluation: things to think about What are your evaluation questions – what do you want to know? A bit on reporting

Inspire Research Ltd Questioning the project How well was the project implemented? To what extent were the objectives met? What were the barriers and enablers that made the difference between successful and disappointing outcomes? What else was learned? (why or how did outcomes happen?) Adapted from Davidson, E.J. (2009) Improving evaluation questions and answers: Getting actionable answers for real-world decision makers. Presented at AEA conference.

Inspire Research Ltd Value or So what? How valuable are the outcomes to stakeholders (students, lecturers, admin staff, the sector)? Was the project worth doing, given the resources used? (Consider the balance of resources to outcomes.) Why would the project, or aspects of it, be worth implementing elsewhere? (How reusable is it?) How sustainable is the project? (Could it continue or expand with limited external funding?) Adapted from Davidson, E.J. (2009) Improving evaluation questions and answers: Getting actionable answers for real-world decision makers. Presented at AEA conference.

Inspire Research Ltd Saving a million How do they know? Average cost of commercial alternatives to MyStudyBar = £115 Estimated saving based on MyStudyBar downloads, multiplied by ave cost = £1M

Inspire Research Ltd What is valuable about your project? In ten words outline a Benefit to particular stakeholders (learners, lecturers, admin staff, senior managers, the sector)

Inspire Research Ltd Telling the evaluation story Executive Summary - with headings linked to your key evaluation questions (or objectives). Background – contextual issues that influenced the project – what the project evaluated and why, including Key project evaluation questions. Methodology – the evaluation approach and techniques used (& why). Findings/Results - sections that address evaluation questions, summarising what was found, followed by the evidence.

Inspire Research Ltd But! How policy makers & practitioners access research Very short time to access project outputs More likely to trust information received face-to- face (or from peers) Reluctant to pay for access to academic journals (or ‘pay’ time to look for project findings) Access new research mostly via the internet Adapted from Beckmann, K., & Mason, C. (2010). An investigation of the knowledge exchange practices of the end-users of sustainability research.

Inspire Research Ltd And, it is worth remembering Institutional processes are really complicated. ‘Project’-based evidence usually plays a very minor role (in institutional decision-making). It is possible to influence policy/your institution, but you need: –a holistic understanding of the context you are working in, –additional skills, –to really want to do it. Adapted from Young, J. (2008). Strategies to Enhance Research Impact. ODI, London. Retrieved from

Any questions and thank you! Dr Rachel A Harris tel: web: