Evaluation PGCert in Learning and Teaching Module 402: Course Design and Evaluation

Learning outcome Design effective evaluation strategies through a critical comparison of alternative strategies Or What forms of evaluation occur in your module/ programme now What approaches are used in other programmes What are the strengths/weakness of ‘your’ system ◦ In comparison with others and with a model What might you adapt to improve evaluation practice in your module/ programme

Definitions Evaluation is: A process whereby a teacher, group of students, or their institution, find out, on the basis of their own perceptions, the overall quality of a programme, a particular course, or a teacher generally part of a formative process aimed at improving standards. heacademy.ac.uk Who Does what Why

Some key points Evaluation is more than student feedback ◦ But includes student feedback as a key source of evidence ◦ Here student feedback means feedback from students, about the programme, not feedback to students, about their work USA uses assessment to mean evaluation (UK) In general: ◦ Module evaluation tends to focus on student feedback ◦ Programme evaluation should have a wider more systematic approach ◦ You need to consider both levels in the assignment

Why find out more To find out about: Student learning Student difficulties Your teaching ◦ Strengths and weaknesses To engage students in their learning experience To provide evidence for: Your professionalism Quality control ◦ Organisational requirement Morss & Murray 2005

Why find out more What systems are there in your area for finding out about teaching, learning, student perceptions for your module Articulating your own (departmental) practice and listening to what others do is one step toward the assignment's ‘critical comparison’

Evaluation: The system Evaluation is more than student feedback There are many sources of information It can be thought of as a process ◦ Circular Usually driven by the institution ◦ Often applied differently in different departments ◦ Regularly applied to modules & programmes as well as to individual lecturers

The evaluation cycle 5 Agree action Notify students, staff 6 Implement changes Report 3 Gather: Data, feedback 2 Devise: Strategy - method(s), timing 4 Analyse & interpret 1 Decide: Reasons, purpose, level HEAcademy.ac.uk

1 Purpose of Evaluation Main purpose: enhance students’ experience of learning and teaching contribute to monitoring and review of quality and standards Also contributes to: effectiveness of course design and delivery dialogue with students students reflecting on their experiences enhancing the teaching and learning process identifying good practice measuring student satisfaction which can lead to marketing (recent development) staff development heacademy.ac.uk Question: Which of these predominate in your area

‘Level’ of the evaluation - where is the focus? Individual session(s) – delivery, learning Module – delivery, administration/resources Programme – delivery, integration, overall University – services Critical questions In your area: Which of these is evaluated as a rule How is it done and who causes it to be done Who gets any value

2 Devise the strategy Sources of information, and timing Sources of information: Students: ◦ Feedback ◦ Staff student liaison committee ◦ Performance in assessment ◦ National Student Survey – increasingly important Self, peers, mentors, external examiners, reviews External sources: ◦ Employers and Professional bodies ◦ Other universities’ courses ◦ League tables Question: Which of these has been useful or impacted on your work

When to evaluate - timing To benefit current students:  After an individual session  Within a module - for immediate feedback on comprehension, interest, perception of relevance, workload To benefit future students:  At the end of a module - to gauge whether learning outcomes have been met, and potential changes for the next time  At the end of the year?  Every 5 years? (UK systems) Which of these are you aware of/ benefit from?

3 Gather: How and what process? Method: Questionnaires (hard copy and electronic) Focus group – various approaches Student assessment results Peer review CATs - Classroom Assessment Techniques Any other way? Process: Who will collect this data ◦ When - in a session, after it, ◦ What format – paper forms, summary report, electronic??

Questionnaires: rationale  Decisions to be made: ◦ Quantitative ◦ for comparison between modules/teachers and over time ◦ Qualitative ◦ to allow more in-depth and tailored responses ◦ Open or closed questions ◦ Hard copy or VLE/ web based (Survey Monkey) ◦ Repeating questions from previous years (for comparison) or changing the questions to give new insights ◦ Aligning questions to the NSS

Questionnaires: design  Bear in mind: ◦ Is the purpose clearly stated ◦ Are the questions unambiguous. Are the questions answerable ◦ Are they phrased in a way that is easily understood ◦ Is there a mix of open and closed questions ◦ How many questions are being asked ◦ Is the Likert scale being used appropriately? (adapted from Brennan and Williams 2004) NB questions themselves are often not researched or evaluated See the resource in vital: ‘Questions to consider when critiquing the evaluation questionnaires’

4 Analyse - interpret How are paper responses collated – put into electronic format? How analysed? How are qualitative comments analysed and reported? Who does all this? Question: What happens in your area?

5 Agree action What happens to the results of the evaluation? How do the students or providers of feedback know what will be changed for next time? Where is the commitment to change? Who is responsible for ensuring that change happens? Question: In your area: How does this happen? Does it ‘work’?

Change the delivery Update the curriculum Provide opportunities for staff development Clarify expectations look at the student handbook! Update resources Tell the Students!! 6 Implement the changes Question: Who does all this in your area?

How can we get students to respond and take it seriously? Make time in sessions Require attendance/ completion Make it easy (e.g. Vital) Don’t overload - too many questions, too often Vary evaluation methods Written comments rather than tick boxes? Give examples of constructive feedback Establish ground rules Give feedback e.g. on notice boards, Vital Demonstrate the effect Question: Which of these is used in your area Which could be?

Coming soon…….. University purchase of Evasys ◦ Will change evaluation practice Increasing responsiveness to NSS ◦ Becoming more thoughtful, less reactive?

NSS Questions: areas & e.gs. The teaching on my course Staff have made the subject interesting Assessment and Feedback Feedback on my work has helped me clarify things I did not understand Academic support I have received sufficient advice and support with my studies Organisation & Management The course is well organised and is running smoothly Learning Resources The library resources and services are good enough for my needs Personal development I feel confident in tackling unfamiliar problems Overall satisfaction Overall, I am satisfied with the quality of the course

Possible additions to NSS Academic challenge/reflective and integrative learning, e.g. ◦ My course has challenged me to achieve my best The learning community/ collaborative learning ◦ I have not been encouraged to talk about academic ideas with other students Student voice ◦ Staff appear to value the course feedback given by students I have been encouraged to use technology to enhance my learning HEFCE 2014

Alternatives UK variation on National Survey of Student Engagement; NSSE (US & Aus) ◦ Gibbs (2012): “NSSE provides a more valid predictor of student gains than does NSS” ◦ See HEA website ‘Engagement for enhancement’ & ‘Institutional case studies (NSSE pilot)’

What other processes contribute to evaluation? Recap: the university has other processes beyond student feedback to evaluate the quality of programmes ◦ Annual Subject Review report ◦ Internal Periodic Review ◦ Professional Body Review ◦ External Examiners ◦ Institutional Audit – QAA (November 2015) ◦ Required responses to NSS  see Student Experience Committees Have you experience of these Have you seen value from them

Learning outcome Design effective evaluation strategies through a critical comparison of alternative strategies Or What forms of evaluation occur in your module/ programme now What approaches are used in other programmes What are the strengths/weakness of ‘your’ system ◦ In comparison with others and with a model What might you adapt to improve evaluation practice in your module/ programme

Questions In your department/programme/team how effectively are evaluation processes leading to change Consider one major change that you could make to evaluation in your area What aspects of ‘your’ evaluation systems would you recommend to others Use the questions on the slides as prompts for your assignment

Resources Angelo, T., & Cross, P. (1993) Classroom Assessment Techniques: A Handbook for College Teachers, 2 nd edition Brennan, J & Williams, R.(2004) Collecting and using student feedback First words on Evaluation- Oxford Brookes University Haugen, L, (1999) Classroom Assessment Techniques (CATs), Iowa State University HEFCE (2014) NSS Review Macdonald, R. (2006) The use of evaluation to improve practice in learning and teaching. Innovations in Education and Teaching International Vol. 43, No. 1, February 2006, pp. 3–13 Morss, K., & Murray, R. (2005) Teaching at University: A Guide for Postgraduates and Researchers Shevlin M., Banyard P., Davies M., & Griffiths M. (2000) The Validity of Student Evaluation of Teaching in Higher Education: love me, love my lectures? Assessment & Evaluation in Higher Education, Volume 25, Number 4, 1 December 2000, pp (9) Assessment & Evaluation in Higher Education

Why find out more Demonstrating evidence based practice Improving student experience Improving your experience! Raising profile (yours) Supporting change

One-Minute Paper Course Knowledge and Skills During last few minutes of class period, ask students to use a half-sheet of paper and write "Most important thing I learned today and what I understood least." Review before next class meeting and use to clarify, correct, or elaborate. Haugen 1999 (from Angelo & Cross 1993)

Muddiest Point Course Knowledge and Skills Similar to One-Minute Paper but only ask students to describe what they didn't understand and what they think might help. Same as One-Minute Paper. If many had the same problem, try another approach. Haugen 1999 (from Angelo & Cross 1993)

How can you gather evidence And what counts as evidence Your own evidence: Teaching diary: your record, responses, reflections, rationales, observation of students, informal notes (e.g. on ppt slides) Video or audio taping Checklist Learning log ◦ Goals and achievements (may be shared)

Various Teaching methods Questionnaires ◦ Standardised ◦ Designed by you for specific purposes ◦ Input from students Focus groups ◦ Run by neutral person Staff- student liaison committees Peer review ◦ Peer observation; mentor; supervisor