Evaluating your project Going beyond student satisfaction Ivan Moore RAEng, Eng SC

Evaluation Why evaluate? Minimise risk Partnership with students Professional practice Publication

Theory Developing teaching practice

Theory Developing teaching practice scholarship

Previous experienceTheory Developing teaching practice scholarship

Previous experienceTheory Different contexts Developing teaching practice scholarship

Previous experienceTheory Different contexts Teaching in practice Developing teaching practice scholarship

Previous experienceTheory Different contexts Teaching in practice Outcome Developing teaching practice scholarship

Previous experienceTheory Different contexts Teaching in practice Outcome Reflection, feedback Developing teaching practice scholarship Further scholarship

Principles of evaluation An integral part of our teaching practice An ongoing process, so that we learn from systematic reflection Should be participatory Should enable us to make appropriate modifications along the way Should enable us to make judgments on specific sessions, but also to draw out wider implications

Purposes of evaluation Mike Prosser Quality Assurance –student satisfaction –the mean score is important as a measure of quality Quality Enhancement –Student conceptions/how they experience the course –The deviation is important: more focused view

Purposes of evaluation Mike Prosser Is the learning environment/teaching approach having any influence on student conceptions/approaches? A student experience survey is more important than a student satisfaction survey

Evaluating your development Three approaches –Goal-oriented evaluation –Goal-free evaluation –Context-based evaluation Several methods –Questionnaire –Focus groups –Structured interviews frequent, large student numbers, superficial summative, representative sample, rich information

Student learning as a journey Starting point The outcomes (or goals) The learning environment

Evaluating the learning journey Expectations and preparedness The outcomes The learning environment

3 Evaluating the outcomes What skills have they developed? –intellectual –professional –inter-personal What other attributes or attitudes have they developed? –confidence –problem-solving –self-awareness (metacognition) Expectations and preparedness The outcomes The learning environment

2 The learning environment How are they experiencing the learning environment? –What opportunities is it providing? –What skills/attitudes do they feel they are developing? –What model or theory are you using? motivation sense of control confidence Expectations and preparedness The outcomes The learning environment

1 Expectations and preparedness Expectations –Standards - challenge –skills and attitudes they will develop –extent of collaboration and independence –role of the tutor Preparedness –meeting demands –working collaboratively –asking questions –active learning –reading and analysing –finding information Expectations and preparedness The outcomes The learning environment

Goal oriented evaluation Determine your goals –4 or 5 at the most For each goal, devise 4 or 5 statements to test achievement or progress –Scored 1-5

Deconstructing an example Find the statements that focus on: 1 Team work 2 Research skills 3 Motivation

Deconstructing a goal Can you identify any theory or model to help you with this? Motivation –interest in the subject/task –clear goals, tasks and standards –support –feedback –sense of belonging and sharing –success - improvement

Designing an evaluation questionnaire In your group Quickly agree and write down some learning experience(s) and outcomes you would like for your students [goals] For each goal Devise 4 statements that will help to determine if the goal is being achieved Record the goals and statements on a flip chart

Goal – free evaluation (Scriven) What did the students get out of the experience? –skills –attitudes –enjoyment –self-awareness

Goal – free evaluation (Scriven) The module has helped me develop my ability to work as part of a team –Is goal oriented. The goal is team working The most useful thing I learned in this module was –Is goal free. Why? –Illuminative evaluation

Goal-free evaluation In your group Devise up to 3 ‘questions’ that you might ask students to answer that might provide information on what they are experiencing Record your questions Discuss what information you might receive from these questions Draw up your conclusions for feedback Share our outcomes

Context-oriented evaluation Where, when and how are the students gaining the experiences and outcomes you wish for them? –(may not be within your module) Since the beginning of the year/module: What skills, if any, have you developed? What helped you to develop these skills? How have you changed the way you study? What prompted you to make these changes?

Context-oriented evaluation Since the beginning of the year: What skills, if any, have you developed? What helped you to develop these skills? How have you changed the way you study? What prompted you to make these changes?

Other methods Focus groups Structured interviews Continuous feedback –What did you find most difficult/confusing today? –What things did you find helped you learn last week? –What should I: Start/stop/continue?

The three minute paper What was the most useful or meaningful thing you learned during this session? What question(s) remain uppermost in your mind as we end this session? What was the ‘muddiest’ point in this session? What would you like me to stop doing? What would you like me to start doing? What would you like me to continue doing?

Other instruments Kolb’s Learning Style Inventory (LSI) Honey and Mumford’s LSQ Sternberg’s thinking styles Felder’s Index of Learning Styles Weinstein’s Learning and Study Strategies Inventory (LASSI) Entwistle’s Approaches to Study Inventory

The Course experience questionnaire (P Ramsden) Designed as a performance indicator 24 statements relating to 5 aspects 1 overall satisfaction statement Research-based Drawn from statements made by students in interviews Students with positive responses take a deep approach

The five sub-scales Good teaching Clear goals and standards Appropriate assessment Appropriate workload Generic skills

The five sub-scales Good teaching3, 7, 15, 17, 18, 20 Clear goals and standards1, 6, 13, 24 Appropriate assessment8, 12, 16, 19 Appropriate workload4, 14, 21, 23 Generic skills2, 5, 9, 10, 11, 22

From Ramsden CEQ The module has helped me develop my ability to work as part of a group I have usually had a clear idea of where I was going and what was expected of me I have found this module interesting The tutor motivated me to do my best work Learning this way has helped sharpen my analytical skills There has been more assessment of what I have memorised than of what I have understood This module has helped me to develop the ability to plan my own work