1. Evaluating E-Learning Efficacy for blended learning courses : University of York, UK Wayne Britcliffe ALT-C : A confrontation with reality 11 th September 2012 What’s going on?

2. Drivers  Initially approached by our Department of Health Sciences: –Health Sciences have extensive and embedded online support for teaching activity –Large student numbers and a varied portfolio including CPD courses as well as UG and PG courses –Devolved VLE administration/acknowledged VLE Coordinator  Approach backed by Health Sciences senior management –Their primary goal was to establish the level of online engagement from staff and students and also try and get an idea of its impact

3. Evaluation challenges  Students range from Undergraduates on campus through to CPD distance learners  Large number of modules to ‘audit’ (circa 217)  How to broadly classify and indicate activity without being judgemental about the effectiveness of that activity?  Establishing actual impact on learning or quantifying where value has been added can be very difficult  E-Learning evaluation not built into course design and the focus of evaluation endeavours quite wide  Difficulty in getting good survey (survey fatigue) and focus group coverage across the whole offering

4. Process – Step 1  Consultation –Departmental VLE coordinator liaised with departmental management  Establish reporting requirements  Get senior backing for survey/report activity –E-Learning Team liaised with the departmental VLE coordinator to develop evaluation plan  Scoping of evaluation aims  Identify useful review and performance indicators  Establish data collection methods

5. Evaluation plan Focus Establish level of on-line activity per course focussed around usage of each course’s discrete VLE module Try and establish the impact of the on-line support/activity on learning Key Questions How has each VLE module been used by teaching staff? How has each VLE module been used and received by the students? Stakeholders For feedback : The department’s teaching staff, the department’s students For evaluation : ELDT, dept’s support staff and management Time Scale/Dependencies Need established late 2011 HS draft report required by mid-March 2012 Instruments and methods Statistical querying of the VLE database Survey of staff users Survey of student users Focus groups with students Systematic audit of current VLE sites

6. Process – Step 2  Data Capture –Statistical querying of the VLE database  VLE administrator written queries based on stat data that is deemed useful –Survey of staff users  Develop survey questions –Survey of student users  Develop survey questions –Focus groups with students/staff  Develop focus group questions –Systematic audit of all current academic year on-line sites  Decide on ‘positioning’ criteria and also on what to qualitatively look at

7. Data Capture – Closer look at module audit – 3E  Systematic audit of all current academic year on-line sites

8. Enhance – Health Sciences Example

9. Extend – Health Sciences Example

10. Empower – Health Sciences Example 1

11. Empower – Health Sciences Example 2

12. Process – Step 3 – Analysis and recommendations EARL (reading lists)

13. Process – Step 3 – Analysis and recommendations  Some survey feedback from students: “Better training for teaching staff, so they are able to offer the different opportunities for as example group files/file exchange/place to save articles and other documents to make them accessible to the whole group. This would decrease the amount of emails sent between groups and make communication easier.” “The Wiki section of the VLE could use clear directions on conventions and getting started.” “I don't consider myself computer illiterate, however The VLE in my opinion is extremely difficult to negotiate.”

14. Process – Step 3 – Analysis and recommendations  Some survey feedback from staff: “Departmental strategy for implementation of VLE in teaching should be developed (and adhered to) - based on needs/requirements of both staff and students - this could (should) be tied into staff development so that awareness of what the VLE can do and how it can support their pedagogical framework and desired course outcomes is increased.” “VLE sites are generally set up by someone else, and I'm not sure if they consult about structure, nomenclature or branding.” “I find it difficult to invest in developing and maintaining the sites due to workload and technical skills.”

15. Process – Step 3 – Recommendations  Staff support and training/student support –Target programmes rather than individuals to help improve understanding/integration of E-learning. Improve student induction to on-line tools/content and support conduits  Template development –Develop templates that suit specific programmes (not one size fits all) and make sure staff understand how to use them  Reading Lists –Clearly mandate Reading List use (for modules that have/require one). Use EARL.

16. Process – Step 3 – Recommendations  Responsibility and communication –Ensure staff know who to turn to for support and that those staff are clear on their responsibilities and what portion of their time should be allocated to such activities  Develop E-Submissions activities –The audit revealed little formative and summative assignment submission. Work is, however, already going ahead on developing E-submissions workflows within the Department  Improve dissemination of good practice –Nurture E-learning champions and improve dissemination/acknowledgement of good practice

17. Moving forward  Refine evaluation processes –Fine tune survey questions/focus group questions –Create a transferable framework  Build evaluation into module design so it’s easier to establish actual impact  Further develop the direct use of 3E in working with staff to further their integration of E-Learning in teaching practice

18. Questions? wayne.britcliffe@york.ac.uk Wayne Britcliffe University of York, UK

19. References Smyth et al., 2011, Benchmark for the use of technology in modules, Edinburgh Napier University Higher Education Academy (2008), Challenges and Realisations from the Higher Education Academy/JISC Benchmarking and Pathfinder Programme: An End of Programme Review by the Higher Education Academy, Evaluation and Dissemination Support Team, September 2008, available at http://elearning.heacademy.ac.uk/weblogs/pathfinder/wp- content/uploads/2008/09/Bench_and_PathFinalReview20080926.pdf http://elearning.heacademy.ac.uk/weblogs/pathfinder/wp- content/uploads/2008/09/Bench_and_PathFinalReview20080926.pdf ACODE (2007), ACODE benchmarks for e-learning in universities and guidelines for use, June 2007, available at http://www.acode.edu.au/resources/acodebmguideline0607.pdf http://www.acode.edu.au/resources/acodebmguideline0607.pdf