1. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution- Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark).Creative Commons Attribution- Noncommercial-ShareAlike 2.0 UK If you adapt this resource for your own use please mark it as a derivative work of the original. The licence information in the footer and University of Bath logos can be amended in the Slide Master (View>Master>Slide Master)

2. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) PG Cert in Academic and Professional Practice e-Learning Unit Session 4 Evaluation

3. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Learning Outcomes design an evaluation plan including formulating an appropriate research question appreciate the pros and cons of a range of data collection methods (and methods of analysis) identify and take account of the ethical issues that may arise from your proposed evaluation select relevant e-learning research literature to inform your practice in HE

4. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) ‘Learning technologies are not transparent, their processes not obvious and they do not broadcast their utility' (Salmon, 2005) This image has been made available on Flickr by Rich_Lem under a CC BY NC SA Licenceimage Rich_LemCC BY NC SA Licence This image has been made available on Flickr by hermida under a CC BY NC SA Licenceimage hermida CC BY NC SA Licence

5. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) “Without asking hard questions about learning, technology remains an unguided missile” (Ehrmann, 1995) This image has been made available on Flickr by US Navy under a CC BY Licenceimage US Navy CC BY Licence

6. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Messiness… It is difficult to separate a specific e-learning intervention from the complex interplay of cultural and social influences on the learning experience. This image has been made available on Flickr by bywaters under a CC BY NC Licenceimage bywaters CC BY NC Licence (Learning and Development Centre, University of Warwick)

7. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Task – 1 minute plans 1 – Describe any plans you have for your e-learning activity to the whole group. 2 – In pairs or threes with similar designs/technologies or context discuss the following questions: What outcomes are you seeking from your learning activity? What do you really want to know? What aspects of the activity do you want to evaluate?

8. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Where is your focus? pedagogical aims learning and teaching processes particular aspects of the learning environment. technology itself, the support surrounding the use of the technology.

9. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) What is evaluation?

10. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) “A good evaluation will have explicit aims, address questions that are meaningful to these aims and to the intended audience, and use an appropriate mix of methods for gathering, analysing and presenting data.” (Learning and Development Centre, University of Warwick) What is evaluation?

11. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Purposes of evaluation diagnostic – learning from the potential users; to inform plans to change practice; formative - learning from the process; to inform practice; summative - learning from the product; to inform others about practice.

12. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Action Research This slide previously contained a diagram of the Action Research cycle.

13. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Action Research Barrett and Whitehead (1985) ask six questions to help start an inquiry: What is your concern? Why are you concerned? What do you think you could do about it? What kind of evidence could you collect to help you make some judgement about what is happening? How would you collect such evidence? How would you check that your judgement about what has happened is reasonable, fair and accurate?

14. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Your question(s) “If you don't have a question you don't know what to do (to observe or measure), if you do then that tells you how to design the study” Draper (1996)

15. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) What is the nature of student engagement? How does the effectiveness of the approach in learning gains compare with its efficiency in terms of teaching efforts (a form of cost-effectiveness)? How confident were female students in discussing their own ideas online compared to their male counterparts? Does it work? Do students like it? Is easy to use? ? This image has been made available on Flickr by wheatfields under a CC BY Licenceimage wheatfields CC BY Licence

16. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Task – think of a question(s)… A few examples: What do tutors/students think of the use of the wiki for a group work activity? What are the perceived benefits and limitations? How has the forum discussion activity helped or hindered students achieve the learning outcomes (for the activity/unit)? What impact has the role play activity had on student motivation? How does an EVS supported activity promote deep learning? How does a ‘Moodle workshop’ mediated peer assessment activity add or detract from the student learning experience? How has formative feedback been improved through the use of a group project wiki? How and when do students use a self-test quiz activity to support their learning on the unit?

17. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Methods of Data Collection Surveys – post/pre-task survey (prior experience and motivation) Focus group/individual interviews VLE access logs Reflective diaries Observation and/or video recording (e.g. Panopto) A learning test (quiz) Exam (assessment) performance Recommended resource: Evaluation cookbook http://www.icbl.hw.ac.uk/ltdi/cookbook/contents.html

18. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Example “recipe” Project Aim: To develop on-line self assessment materials for Y1 UG Biology large enrolment class. Evaluation purpose: To establish student use of and attitudes to on-line self assessment materials. Methods: Access logs, post activity survey (all students), post exam focus group (6 students, refreshments provided)

19. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Ethics is easy? Do no harm Be open Be honest Be careful Shank (2002)

20. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Task Discuss the ethics scenarios and identify any ethical issues that could arise.

21. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Key concerns for this Unit Tutor’s power over students possibly influencing disclosure What can be done to minimise this? Informed consent What would you include on a consent form?

22. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) “Satisfactorily completing an ethics form at the beginning of a study and/or obtaining ethics approval does not mean that ethical issues can be forgotten, rather ethical considerations should form an ongoing part of the research.” Miller & Bell (2002)

23. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) Where next? Key dates Get feedback on your draft design brief –date. Participants’ 5 minute presentations (get peer feedback and support) – date Review session – date Final Submission – date

24. This resource has been released by the University of Bath as an Open Educational Resource. The materials are licensed under a Creative Commons Attribution-Noncommercial-ShareAlike 2.0 UK licence (except the University of Bath Logo which is a Registered Trademark) References Barret, J. Whitehead, J. (1985), Supporting teachers in their classroom research, University of Bath, School of Education Cousin, G. (2009) Researching Learning in Higher Education. Oxon:Routledge Draper, S. (1996) Observing, Measuring, or Evaluating Courseware. Available on the web: http://www.psy.gla.ac.uk/~steve/Eval.HE.html http://www.psy.gla.ac.uk/~steve/Eval.HE.html Ehrmann, SC (1995) Asking the Right Questions: What Does Research Tell Us about Technology and Higher Learning? In Change, Vol. 27, No. 2 (Mar. - Apr., 1995), pp. 20-27 Learning and development Centre, University of Warwick, E-Learning Guide: Evaluating e-Learning Development [online]. Available from: http://www2.warwick.ac.uk/services/ldc/resource/eguides/elearning/what/ [20 May 2011].http://www2.warwick.ac.uk/services/ldc/resource/eguides/elearning/what/ Miller, J. and Bell, L. (2002) Connecting to what? Issues of access, gatekeeping and “informed” consent, In M. Mauthner, M. Birch, J. Jessop and T. Miller (Eds.), Ethics in qualitative research. London:Sage Oliver, M., Harvey, J., Connole, G. & Jones,A. (2007) Evaluation. In G. Connole & M. Oliver (Eds.) Contemporary Perspectives in E-learning Research. London:Routledge Salmon, Gilly(2005) Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions, ALT-J, 13: 3, 201 — 218 Shank, G.D. (2002) Qualitative Research: a personal skills approach. Columbus, OH: Merrill Prentice Hall