Effective dissemination and evaluation Workshop on Teaching and Learning Grants and Carrick Schemes, 2008 Merrilyn Goos and Clair Hughes m.goos@uq.edu.au; clair.hughes@uq.edu.au

Session overview A project scenario Dissemination and evaluation defined Purposes of dissemination and evaluation Small group development of dissemination and evaluation strategies Resources

A project scenario “Introduction to University Teaching and Learning” program: aims, objectives, participants, teaching team New course structure and content from 2008 How to disseminate this innovation - not just to inform potential participants and other stake-holders but to embed good practice? How to evaluate the course in its first offering in order to monitor and improve its implementation?

Dissemination defined Dissemination is more than distribution of information or making it available in some way. Dissemination also requires that some action has been taken to embed and upscale the innovation within its own context and to replicate or transform the innovation in a new context and to embed the innovation in the new context. Carrick Institute Dissemination Framework

Evaluation defined Evaluation is the process of determining the merit and/or worth of things for a range of purposes. Consider funding agency requirements for evaluation: Ensure that evaluation is undertaken at multiple points throughout the project and is understood and reported within an evaluation framework. Carrick Competitive Grants program: Guidelines and Supporting Information – 2008 (p.23) Must provide a strategy for the evaluation of the project with information on outcomes for students, whether there has been a noticeable improvement or not in the expected outcomes for students and how the project outcomes will be disseminated to others within the discipline. UQ Teaching and Learning Small and Large Grants Scheme Guidelines

Purposes of project dissemination Dissemination for awareness (scattering the seed to the wind) Information provision Distributing information through reports, publishing papers, conference presentations, sending emails, developing websites. One way information process. Dissemination for understanding (sowing the seed in prepared soil) Distributing information in a more purposeful manner, e.g. workshops, visits. Requires interaction with others who are involved in the project. Dissemination for action (propagating, breeding, growing, grafting) Engaged dissemination Purposeful, directed, systematic, proactive engagement, evidence based, involves adaptation and implementation, resulting in changes in practice.

Purposes of project evaluation Formative (to inform ongoing project decision-making) Monitor project progress Revise/enhance project processes and outcomes Engage key stakeholders early and during project to: demonstrate respect intensify participation collaborate in identifying key indicators of success progressively expose to new perspectives, or information Make interim reports Summative (to determine/support project findings and outcomes) Report on success/results/value Demonstrate impact Account to sponsors Sustain project outcomes Support related submissions (eg grants for funding for further related projects; promotion; teaching and learning awards) Enlightenment (to learn) Provide opportunities for project team to maximise the learning developed through participation in the project Adapted from “The Learning Partnership” 2007 and W. K. Kellogg Foundation

Discussion In groups of 3 or 4: address the following dissemination and evaluation questions in the context of the project scenario presented earlier (new “Introduction to University Teaching and Learning” program) Consider how you would address dissemination and evaluation for a project in your own educational context Share your responses with the larger group

Dissemination questions Who are the users of and other stakeholders in this program? What do we want to disseminate (process, ideas, products)? How could we make users and other stakeholders aware and involved? How could we engage users and other stakeholders so they use and/or support the program? How could we obtain feedback from users during the project? How will we know there has been change in practice?

Evaluation questions Who is our project audience (stakeholders)? Select two stakeholders/groups and list one or more questions each may have in relation to this project. How could we collect data to answer these questions? (methods) From whom or where (stakeholders and others) could data be collected? (sources) What will count as evidence? (consider both qualitative and quantitative)

The evaluation framework EXAMPLE Evaluation questions Sources/Methods Institutional contacts Expert reference group Existing records/ artefacts What is Generic Graduate Attribute (GGA) policy in Australian universities? Telephone interview (to clarify, address omissions or current activity) Web search of university policy documents What are current barriers to effective GGA implementation? Focus groups to elicit responses to issues paper based on literature review Written responses to draft issues paper Literature review (research into implementation experience) Analysis of AUQA reports

Models of planning for evaluation Implementation Evaluation Implementation Evaluation Dissemination Dissemination Evaluation Implementation Implementation Evaluation Dissemination Dissemination

References Carrick Institute for Learning and Teaching in Higher Education. (2006). Carrick Institute Dissemination Framework Retrieved from http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/dissemination_carrickframework_2006.pdf Carrick Institute for Learning and Teaching in Higher Education. (2008). Competitive Grants program: Guidelines and Supporting Information – 2008. Retrieved from http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/grants_competitive_guidelines2008_july07.pdf Evaluating projects resource  A resource to assist applicants planning the evaluation framework component of their Grants Scheme application has been prepared by Murdoch University <http://www.murdoch.edu.au/teach/carrick_evaluation/index.html> Felton, P., Kalish, A., Pingree, A., & Plank, K. (2007). Towards a scholarship of teaching and learning in educational development. In D. Robertson & L. Nilson (Eds.), To Improve the Academy: Resources for Faculty, Instructional, and Organisational Development (pp. 93-108). Boston: Anker. The Learning Partnership. (2007). Material distributed during Leading and Managing Projects, a workshop sponsored by the Carrick Institute for Learning and Teaching in Higher Education. Melbourne, 17 – 18 September. TEDI Evaluation Services http://www.tedi.uq.edu.au/evaluations/index.html The University of Queensland. (2007). UQ Teaching and Learning Small and Large Grants Scheme Guidelines. Retrieved 29 October, 2007, from http://www.uq.edu.au/teaching-learning/download/TLStrategicGrantsGuidelines07.pdf W. K. Kellogg Foundation. Executive summary: Introducing Program Logic Models. Material distributed during The Professional Development Program for Medical and Healthcare Educators offered jointly by the University of Queensland and Harvard Medical International. 21-25 August, 27-29 November, 2006.