1. Introduction to Evaluation
West Metro Neighbourhood Houses
Centre for Evaluation and Research
December 2018

2. Acknowledgement of Country
The CER team are honored to acknowledge the rich culture, traditions and elders of Victoria’s Aboriginal community. We acknowledge the Traditional Owners and custodians of the land on which we gather today. Embracing the spirit of reconciliation we recognise and value the ongoing and enriching contribution of Aboriginal communities to Victorian life.

3. Introductions Who are we? Training outline
Gina Mancuso, Jane Howard and Ellie McDonald
The Centre for Evaluation and Research
Victorian Department of Health and Human Services
Training outline
An overview of how to develop a program logic
An overview of how to develop evaluation questions
Evaluation next steps – including data sources and reporting
Link using the program logic model to help develop your evaluation questions and indicators to answer your evaluation questions and the data sources for those indicators
How doing your program logic model helps to clarify what you want to know about your program and how you might measure that – make sure you measure what is appropriate and relevant
Logic models help to make explicit assumptions we have about the program, the people involved, how it will operate
Logic models can help you decide what focus your evaluation questions need to have
So if you need to show what you are doing is effective or how you can improve your program you might want to develop questions that focus of how your program was implemented
Alternately, if you need to demonstrate to your funders that your program is achieving its stated outcomes or aims then your evaluation questions will focus on the outcomes or expected benefits from your program (i.e. changes in behaviour, increased skills, changed attitudes or improved conditions)

4. Components of a program logic model
Inputs – resources necessary to implement activities and outputs – money, staff, volunteers, facilities, equipment, supplies
Activities – linked to outputs (reframe output into an active task) – services, training, education (e.g. provide job training, educate public about healthy eating or exercise, create mentoring partnerships for youth)
Outputs – direct products of program activities (i.e. units of service like hours of service provided or products – number of participants, number of education/training sessions held, number and types of educational or training material distributed, number of clients linked with mentors)
Outcomes – short to medium term (e.g. changes in knowledge, attitudes and skills; changes in actions, behaviours or policies; changes in environmental, social, economic conditions
Assumptions – factors that you take for granted that mean the chain of events takes place
So why we think activity X will lead to output Y and/or outcome Z
Are there other explanations for why activity X might lead to outcome Z?
Will activity X always lead to outcome Z or only under some circumstances and with some target groups?
External factors – factors that may impact the outputs and outcomes (positive or negative) and over which you have little control
These can be funding, policy landscape, changes at local or state government
Demographic makeup, family circumstances, values, political environment, background and experiences of participants, media, policies and priorities
Remember the logic model is just that, a model. It is not intended to portray a program in minute detail, rather it is a way of understanding how a program might achieve its desired outcomes
So a logic model needs to be clear and understandable and best fits the user and use
There is no best model – we learn through trial and error and practice, program logic models may also change as a program evolves

5. The If and Then of logic models
Certain resources are needed to implement your program;
If you have access to them, then you can use them to accomplish your planned activities;
If you accomplish your planned activities, then you will hopefully deliver the amount of product and/or service that you intended;
If you accomplish your planned activities to the extent you intended, then your participants will benefit in certain ways;
If these benefits to participants are achieved, then certain changes in organisations, communities, or systems might be expected.
This is the foundation of logic models and is our attempt to articulate a casual association between what we do and our expected outcomes
Outcomes can be split into short and long term
If you have certain resources, then you will be able to provide activities, produce services or products for targeted individuals or groups. If you reach those individuals or groups, then they will benefit in certain specific ways in the short term.
If the short-term benefits are achieved to the extent expected, then the medium-term benefits can be accomplished.
If the medium-term benefits for participants/organizations/decision-makers are achieved to the extent expected, then you would expect the longer-term improvements and final impact

6. Developing your logic model: Start with your outcome/impact and work back from there
Issue/situation: Describe the problem your program is trying to address
Impact: Specify what you want your program to achieve as an indicator of your program’s success (its impact, long-term outcome)
Outcomes: What short to medium term outcomes need to happen if the impact is to be achieved
Outputs: What outputs are needed to achieve these outcomes – you will need to identify what actual activities will be delivered, who will be engaged (i.e. who is your target group) and what kind of response or interaction will be need from participants if the program is to be successful (example output for an education program might be: X number of target participants completed the course)
Inputs: Be as specific as possible about what elements will make up the program, resources needed (i.e. people, partner organisations, skills, equipment, technology) and what activities
Does it all make sense? Does the logic you have outlined make sense? Are your casual links plausible?

7. Activity: develop your own program logic
Identify a program you would like to evaluate
Begin filling out the columns in your program logic template
Use any program documentation you’ve brought along to complete your program logic
We will come around and answer any questions you have
Start with your outcome and work back from there
Issue/situation: Describe the problem your program is trying to address
Impact: Specify what you want your program to achieve as an indicator of your program’s success (its impact, long-term outcome)
Outcomes: What short to medium term outcomes need to happen if the impact is to be achieved
Outputs: What outputs are needed to achieve these outcomes – you will need to identify what actual activities will be delivered, who will be engaged (i.e. who is your target group) and what kind of response or interaction will be need from participants if the program is to be successful (example output for an education program might be: X number of target participants completed the course)
Inputs: Be as specific as possible about what elements will make up the program, resources needed (i.e. people, partner organisations, skills, equipment, technology) and what activities
Does it all make sense? Does the logic you have outlined make sense? Are your casual links plausible?

8. Definitions: Remember a logic model needs to be clear and understandable and best fits the user and use
Inputs – resources necessary to implement activities and generate outputs –
Examples: money, staff, volunteers, facilities, equipment, supplies
Activities – linked to outputs
Examples: services, training, education
Outputs – direct products of program activities
Examples: units of service (e.g. hours of service provided or products), number of participants, number of education/training sessions held, number and types of educational or training material distributed, number of clients linked with mentors – These can become your indicators
Outcomes
Short to medium term - changes in knowledge, attitudes and skills
Longer term changes in actions, behaviours
Assumptions – underlying theories and beliefs about the program and its context
Why we think activity X will lead to output Y and/or outcome Z?
Are there other explanations for why activity X might lead to outcome Z?
Will activity X always lead to outcome Z or only under some circumstances and with some target groups?
External factors – factors that may impact the program and over which you have little control
Cultural norms
Political climate
Funding
Background and experiences of participants

9. Tea break

10. Developing evaluation questions
Evaluation questions should draw out the information that needs to be collected to effectively measure a program’s contribution to change
How do we develop evaluation questions?
Identify what you want to know
Identify the type of evaluation (formative/process/impact)?
Identify how the questions fit in terms of your program logic model
Consult with program stakeholders
Prioritise most important questions (limit questions)
WHAT IS AN EVALUATION QUESTION?
Evaluation questions should focus on what you want to learn from a particular program e.g. if you want to understand how effective a program was you could ask ‘to what extent did the programs inputs, activities and outputs lead to the desired outcomes?’
Evaluation questions act as a overarching guide for the evaluation, and help you determine an appropriate methodology and data collection methods
It is important not to ask too many questions, as this can create a lot of work and also lose the focus of your evaluation
How do we develop evaluation questions?
Identify what you want to know
Identify the type of evaluation (formative/process/impact)?
Identify how the questions fit in terms of your program logic model
Consult with program stakeholders
Prioritise most important questions (limit questions)

11. Key ways to break down evaluation questions
When developing evaluation questions, it is useful to break them down into the five categories of:
Appropriateness – the extent to which a program is appropriate for specific target population and context
Effectiveness – the extent to which a program has been effective
Efficiency – the extent to which the program has been cost effective
Impact – the extent to which a program is on track to achieve its goal/or has achieved its goal
Sustainability – the extent to which there is funding or policy support for a program beyond the trial period
It can be hard to know exactly what to ask, even if you are clear on what you want to know.
These five categories can be a helpful guide to help you cover off on all the relevant areas.
Typically formative or process evaluations focus on appropriateness, effectiveness and efficiency questions
Outcome evaluations can focus on all five categories if appropriate but don’t need to.
READ OUT EXAMPLES FROM EXAMPLE SHEET

12. Let’s develop some evaluation questions
To practice this process, we’ve provided an example of a program logic on your tables.
Does anyone want to have a go at coming up with an evaluation question for this program?
E.g. if we want to understand the impact of this program, we could ask ‘to what extent did VCT training decrease HIV transmission rates?’

13. Activity: develop your evaluation questions
Use the ‘Evaluation checklist’ to start drafting your evaluation questions
Draw on the handout with example questions for inspiration
We will float around to support development of evaluation questions
Now we’re going to have a go at developing evaluation questions for our own programs using the evaluation checklist and the handout with example questions for inspiration
Start by thinking about what you want to know at the end of your evaluation and what your stakeholders would like to know. What will you need to report on to your stakeholders?
We will come around and provide guidance and answer any questions you have

14. Overview of data sources
What types of data will help you answer your evaluation question and provide evidence about whether your indicator(s) has been met?
Think about a range of data sources:
Monitoring data including attendance rates, demographic information
Quantitative data from surveys
Qualitative data from interviews, focus groups
Documentary data from annual reports, case notes,
student progress reports, student work/output
On your evaluation checklists you will see columns to continue planning your evaluation. These include identifying your relevant stakeholders, identifying indicators e.g. how you will know if your program is working, and data sources.
Indicators sound like a tricky one but really they are quite simple. Indicators are measurements or values that tell you whether something is occurring. Use your outputs from your program logic as a guide for indicators. E.g. an indicator could be that 60 people participated in HIV training – this indicates that the training was attended by a high number of people and suggests that more people are better educated, which the program hopes will decrease HIV transmission rates.
Data sources is a good next step. Looking at each evaluation question, how will you best answer it? With what data source?
It’s good to keep in mind what data is already available from the program and what data collection and data sources are feasible to draw on within the timeframes of the evaluation.
Look at your list of key stakeholders when thinking about data sources and think about the range of information and opinions that would be relevant to explore and document.

15. Next steps… Now that you have started developing your:
Program logic and;
Evaluation questions
The evaluation report template will support the continuation of your evaluation planning. The template provides guidance and prompts for areas including:
Methodology
Data collection and analysis
Stakeholder engagement
Governance
Ethical considerations

16. Thank you
Any questions?