Principles of Effective Evaluation Prof Keithia Wilson Academic Leader Student Success & Retention

Importance of Evaluation Results in evidence-based practice Results in evidence-based practice Helps us to identify the most effective & strategic interventions (make a difference or add value) Helps us to identify the most effective & strategic interventions (make a difference or add value) Leads to sustainable & enjoyable practice Leads to sustainable & enjoyable practice Provides accountability to School & Group leadership Provides accountability to School & Group leadership Contributes to scholarship of the FYE Contributes to scholarship of the FYE

Kirkpatrick (1994) Model of Evaluation - Overview 4 levels of outcome evaluation – 1. Level 1 – Reaction 2. Level 2 – Learning 3. Level 3 – Behaviour 4. Level 4 – Results

Level 1 - Reaction Measures participant’s reactions to the training program, including:  reactions to the overall program (outcomes) e.g., * “To what extent did you find the Orientation program useful?” * “To what extent did you find the Orientation program useful?”  reactions to specific components of the program (processes) e.g., * What aspects of Orientation did you MOST APPRECIATE and found USEFUL? * What aspects of Orientation did you MOST APPRECIATE and found USEFUL? * What aspects of Orientation did you LEAST APPRECIATE and feel is MOST IN NEED OF IMPROVEMENT? * What aspects of Orientation did you LEAST APPRECIATE and feel is MOST IN NEED OF IMPROVEMENT?

Level 1 - Reaction Important to include closed-ended items including rating scales e.g., Important to include closed-ended items including rating scales e.g., How effective was the Orientation program overall in helping you to make a transition to university? How effective was the Orientation program overall in helping you to make a transition to university? not at all effective very effective not at all effective very effective How useful was the initial information session on the predictors of success in first year? How useful was the initial information session on the predictors of success in first year? not at all useful very useful not at all useful very useful As well as open-ended items As well as open-ended items What else would you have liked included in the Orientation program today? What else would you have liked included in the Orientation program today?

Level 1 - Reaction Some FYE examples – Survey questions on Orientation (retrospective student survey) Survey questions on Orientation (retrospective student survey) School specific Orientation Program evaluation at the end of O-Day & the Common Time (ongoing Orientation Program) – student surveys, focus groups School specific Orientation Program evaluation at the end of O-Day & the Common Time (ongoing Orientation Program) – student surveys, focus groups Peer Mentoring Program final evaluation by mentors & mentees Peer Mentoring Program final evaluation by mentors & mentees

Level 2 - Learning Measures what participants have learned from involvement in the program What is measured needs to relate to what was covered in the program viz. learning objectives What is measured needs to relate to what was covered in the program viz. learning objectives Typically covers knowledge, skills, or attitudes Typically covers knowledge, skills, or attitudes Needs to include both rating scales & open-ended questions Needs to include both rating scales & open-ended questions Can include self-report & tests of actual knowledge Can include self-report & tests of actual knowledge

Level 2 - Learning Some examples – 1. Student self-report * I feel confident that I know what to do to be successful academically * I feel confident that I know what to do to be successful academically this semester this semester not at all confident very confident not at all confident very confident * What have you learned about the transitions you will need to make as * What have you learned about the transitions you will need to make as a commencing student in order to be successful? a commencing student in order to be successful? 2. Student knowledge 2. Student knowledge * I know the name of my First Year Advisor yes no (circle one) * I know the name of my First Year Advisor yes no (circle one) * I know the difference between a lecture and a tutorial (circle one) * I know the difference between a lecture and a tutorial (circle one) Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree

Level 2 - Learning Some FYE examples – Survey data evaluates learning outcomes in terms of knowledge & attitudes (e.g., knowing where to go to get help; perceptions of good teaching) Survey data evaluates learning outcomes in terms of knowledge & attitudes (e.g., knowing where to go to get help; perceptions of good teaching) Outcome evaluation for Peer Mentees (What are your 3 most important learnings from your participation in the Peer Mentoring Program?) Outcome evaluation for Peer Mentees (What are your 3 most important learnings from your participation in the Peer Mentoring Program?) First-Assessment-First-Feedback amber alert intervention (What have you learned as a result of your participation?) First-Assessment-First-Feedback amber alert intervention (What have you learned as a result of your participation?)

Level 3 - Behaviour Measures the transfer of knowledge, skills & attitudes from the training context to in vivo or real-life contexts (the classroom & study). Evaluate both before & after the program if practical Evaluate both before & after the program if practical Use survey, focus groups, interviews with students, mentors, staff (convenors, sessionals) Use survey, focus groups, interviews with students, mentors, staff (convenors, sessionals) Survey (Did you attend Orientation? How much time are you spending on study?) Survey (Did you attend Orientation? How much time are you spending on study?)

Level 3 - Behaviour Some FYE examples – survey data survey data Attendance at E-Day, O-Day & Common Time/ASP Program Attendance at E-Day, O-Day & Common Time/ASP Program Attendance at lectures & tutes Attendance at lectures & tutes Assessment submission rates in courses Assessment submission rates in courses

Level 4 - Results Measures “return-on-investment”, or the extent to which the program has produced results. Some FYE examples include – Hard student outcomes such as - Hard student outcomes such as - * pass rates in courses for assessment items * pass rates in courses for assessment items * GPA in semester 1, 2 & FY overall * GPA in semester 1, 2 & FY overall * student retention into year 2 * student retention into year 2 Soft student outcomes such as – Soft student outcomes such as – * student satisfaction * student satisfaction

Evaluation Key question – What difference do our FYO&E interventions (e.g., Orientation, Peer Mentoring, Attendance, First Assessment Intervention etc.) make overall? Program/Intervention  1 Reactions  2 Learning  3 Behaviour  4 Overall Productivity/Result Program/Intervention  Feel  Know  Do  what is the result or difference overall?