1. Rebecca McQuaid www.rmcqconsult.com rebecca.mcquaid@gmail.com
Evaluation 101
…a focus on Programs
Rebecca McQuaid

2. About me… Having an “evaluation brain” means… Being curious!
Asking questions
Paying attention
Reflecting
Challenging our beliefs/ assumptions

3. About you… What do you want to take away?
Exercise: Interests & Challenges
This presentation WILL address:
Program Evaluation
Why, what and when… and some of the how
Skims the surface – Backgrounder, 101
This presentation will NOT address:
Developmental Evaluation
Reporting (e.g., How to “tell the story”)
Capacity (Time, money, staff)
What do you bring to the table?
Experience with eval…

4. Eval 101 This presentation WILL address: Program Evaluation
This presentation will NOT address:
Developmental Evaluation
Reporting (How to “tell the story”)
Capacity (Time, money, staff)
This presentation WILL address:
Program Evaluation
Why, what and when
… and some of the how
Backgrounder/ Refresher?

5. EVALUATE … & INCREASE IMPACT!
Why?
*Source: Evaluation for Organizational Learning: Basic Concepts and Practical Tools (Learning for Action Group)
EVALUATE … & INCREASE IMPACT!

6. At its heart, evaluation answers… What is working? 2) What is not?
Helps determine…
How a program is working
How successful a program is in achieving the desired outcomes
The impact a program has on the target group
Level of satisfaction* with the program
Program strengths and weaknesses
Cost effectiveness and efficiency of operation
At its heart, evaluation answers…
What is working?
2) What is not?
3) What difference is being made?

7. Program Evaluation Goals & Objectives (“Why”) Activities (“What”)
Anticipated Outcomes
Actual Outcomes
Conclusions (for learning!)
Summarizes…

8. It can tell us… Is a program working? Achieving outcomes
Level of satisfaction
Strengths & weaknesses
Cost effectiveness
Efficiency
Impact on intended audience
Summarizes…

9. Types of Eval Formative How?
Begins during development and continues throughout program
How?
Needs Assessments
Process Evaluation
Summarizes…

10. Needs Assessment Takes place before program
Is there a need?
What is it?
First step in planning/ design
One-time analysis
Summarizes…

11. Process Evaluation Takes place during program
Assesses how program outcome(s) achieved
Collects data to improve program for efficiency & results
Summarizes…

12. Helps determine…
Are processes used to achieve goals and objectives effective?
Was program carried out as planned?
What worked? Improvements?
Participant satisfaction with program
Summarizes…

13. Types of Eval Summative How? Outcomes-Based Evaluation
Done at end of program
Focuses on success & effectiveness of program in reaching the stated objectives
How?
Outcomes-Based Evaluation
Summarizes…

14. Outcomes-Based Eval Were (pre-determined) outcomes met?
After implementation
Two important points…

15. Helps determine… Impact on intended audience/ population?
Success in achieving outcomes?
Is there evidence to support continuation? Expansion?

16. PSA: Timing matters!
“It’s much more effective to build evaluation in up front during the program planning process – this ensures that you gather the right data at the right time”

17. Types of Eval – Which one?
Formative
Needs Assessments
Process Evaluation
Summative
Outcomes-Based Evaluation
Consider… type of program, purpose of eval, resource availability (people, $$), program timeframe

18. Ready to Evaluate? CREATE A… Tools Program Profile Program Profile
Goals and Objectives
Inputs and resources
Program activities
Target audience/ populations
Tools
Program Profile
Logic Model
CREATE A…

19. Logic Model … evaluable? Visual of how a program (theoretically) works
Shows cause-and-effect relationship between activities and outcomes
Any problems or gaps in program?
… evaluable?
Outputs - Concrete/quantifiable results of project activities – i.e. number of people that attended a workshop, reports, training tools developed, number of promotional activities carried out, etc.
Outcomes - Actual impacts/changes as a result of the project.

20. Logic Model For: Individual programs Specific Detail
Activities, Outcomes
Implementation tool

21. Theory of Change For: Organizations, initiatives High-level overview
Strategies & Outcomes
Guiding/ Communication tool

22. Logic Model Common logic model mistakes:
Activities are not action-oriented or specific enough to really describe what is
being done.
Relationships between activities and outcomes are unclear.
Outcomes are not specific or measurable.
Outputs and outcomes are confused. **
Outputs - Concrete/quantifiable results of project activities – i.e. number of people that attended a workshop, reports, training tools developed, number of promotional activities carried out, etc.
Outcomes - Actual impacts/changes as a result of the project.
*Source: Evaluation for Organizational Learning: Basic Concepts and Practical Tools (Learning for Action Group)

23. Overarching goal: To decrease the rate of unemployment among young adults (aged 18-24) in the City of Kingston.
E.g., Create posters for college & university campuses advertising career counselling program services
E.g.,Two part-time staff with dedicated hours to young adult employment counselling
E.g., Reduce avg. amount of time young adults spend in career counselling programs by 4 months
E.g., Increase 1-on-1 career advising sessions with young adult clients to at least twice a week
Decreased rate of unemployment among young adults in Kingston
Common logic model mistakes:
Activities are not action-oriented or specific enough to really describe what is
being done.
Relationships between activities and outcomes are unclear.
Outcomes are not specific or measurable.
Outputs and outcomes are confused. **
Outputs - Concrete/quantifiable results of project activities – i.e. number of people that attended a workshop, reports, training tools developed, number of promotional activities carried out, etc.
Outcomes - Actual impacts/changes as a result of the project. M

24. (Short term) Outcomes OBJECTIVES
Appreciate the role of evaluation in organizational learning
Understand different evaluation approaches, what they are for & when they are used
Use a Logic Model template to begin to organize a “picture” of your program
Note:
Objectives would be phrased: To appreciate the role of evaluation in organizational learning…
Changes that occur as a direct result of the program activities/strategies over the short-term
Are developed from the program objectives

25. (Long term) Outcome GOAL
E.g., Develop capacity for participants in “Evaluation 101” workshops to plan theory-based program evaluations
Broader level changes that the program is working toward and hopes to achieve/contribute to over the long-term
Developed from the program goals

26. Evaluation Framework Outlines: What will be monitored
How success will be measured
Who/where data will be collected from
How data will be collected
Summarizes…

27. Data Collection Methods: Common ones:
Depend on the type of data that needs to be collected and who data is being collected from
Summarizes…
Common ones:
Document review, Focus groups, Interviews, Surveys, Observation, Tracking/monitoring

28. Eval Framework (cont’d)
Data collection tools are derived from the evaluation framework
Develop a data collection plan
Where will you get information?
When will it be collected?
From whom? How many people?
Etc.
Consistency = reliability
Analyze data & interpret results
Share results
Summarizes…

29. (Short term) Outcomes OBJECTIVES
Appreciate the role of evaluation in organizational learning
Understand different evaluation approaches, what they are for & when they are used
Use a Logic Model template to begin to organize a “picture” of your program
Note:
Objectives would be phrased: To appreciate the role of evaluation in organizational learning…
Changes that occur as a direct result of the program activities/strategies over the short-term
Are developed from the program objectives

30. Rebecca McQuaid www.rmcqconsult.com rebecca.mcquaid@gmail.com
Evaluation 101
…a focus on Programs
Rebecca McQuaid