Impact and Evaluation Impact of Foundations Programs - UWA & literature Impact of Foundations Programs - UWA & literature Implementation / Transfer of Learning to Practice Implementation / Transfer of Learning to Practice Evaluation of programs Evaluation of programs

Limited studies Limited studies Generally single institution / program. Generally single institution / program. Teaching and learning improves Teaching and learning improves (Coffey & Gibbs, 2000; Hall, 1996) Positive teaching evaluations Positive teaching evaluations (Andresen, 1995; Nasr, Gillet & Booth, 1996) Long term positive effects Long term positive effects (Giertz, 1996; Rust, 2000)

Needs a work culture that encourages change and risk taking and values teaching. Needs a work culture that encourages change and risk taking and values teaching. (Spafford Jacob & Goody, 2002) Need for post-program contact between educational developer and participants and their peers. (Spafford Jacob & Goody, 2002) Need for post-program contact between educational developer and participants and their peers. (Spafford Jacob & Goody, 2002) Need to provide systematic induction to teaching as part of graduate education. Need to provide systematic induction to teaching as part of graduate education. (Schulman, 1995)

Foundations of University Teaching and Learning Development of high quality teaching & learning at the university. develop a professional & effective approach to teaching based on critically reflective practice; develop a professional & effective approach to teaching based on critically reflective practice; provide support and resources for participants in their development as tertiary teachers; provide support and resources for participants in their development as tertiary teachers; a forum to discuss issues of teaching & learning; a forum to discuss issues of teaching & learning; develop networks for the continuing develop networks for the continuing dialogue & support; dialogue & support;

2.5-day workshop 2.5-day workshop seven 2-hour follow-up sessions seven 2-hour follow-up sessions online module online module peer observation of teaching component. peer observation of teaching component. Mainly new academics but also experienced & future academics Mainly new academics but also experienced & future academics

UWA – Foundations of University Teaching and Learning UWA – Foundations of University Teaching and Learning (Spafford Jacob & Goody, 2002) (Spafford Jacob & Goody, 2002) surveyed seven foundations programs surveyed seven foundations programs 32 responded (40% response rate) 32 responded (40% response rate) plus information from end-of-program feedback plus information from end-of-program feedback 45% arts, humanities, social sciences 45% arts, humanities, social sciences 55% science, engineering, agriculture 55% science, engineering, agriculture 50% had no prior ‘teacher training’ 50% had no prior ‘teacher training’ majority were relatively inexperienced teachers majority were relatively inexperienced teachers

Outcomes –Positive outcomes – increased theoretical understanding and more teaching and learning skills and methods; –More experienced teachers benefited less; –Reflection is important – although some participants say too much reflection and not enough information; –Networks were established; –Need for focus on content and process relevant to individual’s teaching / discipline context.

Has participation in Foundations enhanced the quality of your teaching practice?

Have you been able to influence the quality of teaching in your school?

Have you been able to overcome the barriers to implementing new teaching practice?

Factors impeding the transfer of learning Impact of organisational culture. Impact of organisational culture. Low value placed on teaching (perceived & real). Low value placed on teaching (perceived & real). Lack of school support for implementation of learning. Lack of school support for implementation of learning. Not enough resources (time, financial, staff) to develop and implement new teaching and learning strategies due to increasing workload and content tyranny. Not enough resources (time, financial, staff) to develop and implement new teaching and learning strategies due to increasing workload and content tyranny.

Relevance of content and activities for individual teaching context. Relevance of content and activities for individual teaching context. Colleagues resistance to change, lack of dialogue about teaching. Colleagues resistance to change, lack of dialogue about teaching. Uniqueness of each individual’s engagement with the program. Uniqueness of each individual’s engagement with the program. Student resistance to new teaching styles. Student resistance to new teaching styles. Physical structure of teaching venues (e.g. fixed chairs and tables). Physical structure of teaching venues (e.g. fixed chairs and tables).

I am rejoicing in having the freedom and control to be able to try some of the things that I learned in Foundations. Here there is less of an established culture of the “way things have always been done”. I am rejoicing in having the freedom and control to be able to try some of the things that I learned in Foundations. Here there is less of an established culture of the “way things have always been done”. (Comments from a Foundations participate who has left the University and works in a newer institution)

Facilitating Transfer of Learning Pay attention to work context that participants will return to. Pay attention to work context that participants will return to. Spend time in the program on transfer / implementation of learning as well as content. Spend time in the program on transfer / implementation of learning as well as content. Involve participants in group discussion to identify barriers to transfer. Involve participants in group discussion to identify barriers to transfer. Include post-activity follow-up as part of program. Include post-activity follow-up as part of program. Need for cooperation between schools and EDU. Need for cooperation between schools and EDU. Involve participants in program planning. Involve participants in program planning.

Evaluation Practices in Canada (Kreber 1997) Most EDUs made some attempt to evaluate parts of programs but comprehensive evaluation was not common; Most EDUs made some attempt to evaluate parts of programs but comprehensive evaluation was not common; Workshops were most frequently evaluated; Workshops were most frequently evaluated; Most only evaluated goals of the programs; Most only evaluated goals of the programs; Few reported assessing program impact; Few reported assessing program impact; Assessment of possible changes in staff’s beliefs or student achievement were the least common evaluation practice; Assessment of possible changes in staff’s beliefs or student achievement were the least common evaluation practice; Most based on only one source of data; Most based on only one source of data;

Most based on informal conversations with participants at end of program – based on likes & dislikes; Most based on informal conversations with participants at end of program – based on likes & dislikes; Systematic analysis of data gathered from evaluations was not common; Systematic analysis of data gathered from evaluations was not common; Experience in educational evaluation seemed to make a difference in how evaluation was approached; Experience in educational evaluation seemed to make a difference in how evaluation was approached; Lack of time, money and staff most common reasons for lack of systematic evaluation; and Lack of time, money and staff most common reasons for lack of systematic evaluation; and More than 65% of developers said systematic evaluation was part of their work. More than 65% of developers said systematic evaluation was part of their work.

Evaluation should address the following questions: What is the intended impact or what are the goals? What is the intended impact or what are the goals? Why evaluate? Why evaluate? When to evaluate? When to evaluate? Who evaluates?; Who evaluates?; How to evaluate? How to evaluate? Is the actual impact the same as the intended impact and is the actual impact desirable?; Is the actual impact the same as the intended impact and is the actual impact desirable?; Who should receive the results of the evaluation? Who should receive the results of the evaluation? What do we expect to gain from the evaluation and what will happen as a consequence? What do we expect to gain from the evaluation and what will happen as a consequence? Kreber & Brook (2001)

Six possible elements to evaluate: Six possible elements to evaluate: 1. participants' perceptions / evaluations; 2. participants' beliefs about teaching & learning; 3. participants' teaching performance; 4. students' perceptions of staff's teaching performance; 5. students' learning; and 6. effects on the culture of the institution. Kreber & Brook (2001)

Aligning evaluation strategies with intervention strategies and level of impact [From: Kreber & Brook, 2001]

Strongest index of effectiveness is impact on students. Weakest index of effectiveness is participants’ self reports. Strongest index of effectiveness is impact on students. Weakest index of effectiveness is participants’ self reports. (Levinson-Rose & Meges, 1981) Goal-free evaluation: rather than focus on goals, focus on outcomes Goal-free evaluation: rather than focus on goals, focus on outcomes (Scriven,1972) (Scriven,1972) Learning is only partially dependent on teaching which makes evaluation of impact even more difficult. Learning is only partially dependent on teaching which makes evaluation of impact even more difficult.