Approaches to Evaluation (1.5) What are the goals of our program? What do we expect to happen to participants? What do we want participants to gain, learn, etc.?

Keep in Mind: Purpose of Evaluation/Research Resources and Skills Available Who Has the Information How Data will be Analyzed and Used

Approaches (Models) Intuitive Judgment Professional Judgment Goal-Attainment Logic Models Goal-Free Process or Systems

Goal-Attainment: Goals and Objectives Related to “needs,” “skills,” and “competencies” Helps to establish a direction for your program Helps to define what resources are needed and how (and where) resources will be used Helps to define what programs/facilities will be provided Specifically links desired outcomes to programs Usually connected to this mission statement of the youth development organization Helps to establish a direction for the program Helps to define what resources are needed and how (and where) resources will be used Helps to define what services will be provided (links customer needs with organizational services) Links benefits to services 4

Goals and Objectives Goals Objectives Broad, long-range statements that define the programs/services that are going to be provided Objectives Specific statements (about the attainable parts of the goal) that are measurable and have some dimension of time Goals can be defined as broad statements that define the youth development program/services that are going to be provided. In contrast, objectives are specific statements that are measurable and have some dimension of time. 5

Objectives (Review) Specific Measurable Achievable Relevant Must be clear and concrete Measurable Must be some way to determine whether or not the desired results have been achieved Achievable Must be attainable and reality-based! Relevant Must be useful; must have worth to your organization Time-limited/Time connected Must specify a time frame for accomplishment Program objectives should have the following five (5) characteristics: One, they should be “Specific”- clear and concrete. Two, they must be “Measurable,” there must be some way to determine whether or not the desired results have been achieved. Three, they must be “Pragmatic,”- which means that they have to be realistic, attainable, and based in reality. Four, they must be “Useful.” In other words, they must have some value, worth, or benefit to the organization or to the youth and families who will be participants in the programs. And five, objectives should be “Linked to Needs,”- they should be directly linked to the needs of youth/families and the organization’s mission. 6

Individual or Group Remember that goals and objectives can be written for an individual (i.e., how you want one particular person to perform/succeed) or a group (i.e., how you want a group of participants to perform/succeed). Recall that our program goal was “to improve youth’s public speaking skills.” By putting all 4 parts of our objective together, we can get the objective: “Youth will demonstrate proficiency in giving an oral presentation by standing in front of a peer group and discussing a selected topic, which has an introduction, main argument, supporting facts, and conclusion in 4-6 minutes.” For this goal, any number of objectives could have been developed. This was just one example. 7

A Logic Model is… A HOT TOPIC in (Program) Evaluation A depiction of a program showing what the program will do and what it is to accomplish A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes The core of program planning and evaluation 8

MODEL LOGIC the principles of reasoning the relationship of elements to each other and a whole MODEL small object representing another, often larger object (represents reality) preliminary pattern serving as a plan tentative description of a system or theory that accounts for all its known properties The American Heritage Dictionary, 2nd ed. Adapted from University of Wisconsin-Extension, Program Development and Evaluation 9

Simplest Form of a Logic Model INPUTS OUTPUTS OUTCOMES In its simplest form, a logic model is a graphic representation that shows the logical relationships between: The resources that go into the program – INPUTS The activities the program undertakes – OUTPUTS The changes or benefits that result – OUTCOMES 10

What will show that you’ve arrived? (EVALUATION!) Where are you going? How will you get there? What will show that you’ve arrived? (EVALUATION!) 11

Logic Model May Also be Called… Program action Model of change Outcome map Program logic 12

Everyday Example Get pills Take pills Feel better H E A D C Situation Let’s take a simple example – one that we can all relate to. How many of us have had a headache at one time or another? (headache – SITUATION) What do we do? Our experience may be that certain pills help So, we need to get the pills (INPUTS), Then we take the pills (OUTPUTS) As a consequence, our headache goes away and we feel better. (OUTCOME) Number of embedded assumption: assumes that we can find/get the needed pills; that we take the pills as prescribed; that the pills lead to improvement – not a stomach ache or other negative side effect. All programs have such assumptions – often the basis for failure or less than expected results But, you can see the logic of the diagram and the end results – the impact that is expected. What really matters isn’t whether we get the pills and take the pills, but whether we feel better as a result Situation INPUTS OUTPUTS OUTCOMES Slide adapted from University of Wisconsin-Extension, Program Development and Evaluation 13

Assumptions Assumptions underlie much of what we do. One benefit of logic modeling is that it helps us make our assumptions explicit. Assumptions are the beliefs, principles, ideas we have about the program, the people involved and the way we think the program will operate. Assumptions underlie all that we do. Examples of assumptions include: Community coalitions are an effective strategy for addressing community problems Our partners will participate actively in program delivery The funding will be adequate and available when needed The target participant want to learn and change their behaviors In a 2004 study by Kaplan and Garrett, assessing underlying assumptions was found to be one of the most important parts of logic modeling but it is often minimized or overlooked. [Kaplan & Garrett (2005)] 14

Youth and Community Service INPUTS OUTPUTS OUTCOMES Youth identify project to work on Youth improve skills in planning, decision making, problem solving Youth demonstrate leadership skills Staff Youth are connected with and feel valued by their community Youth successfully complete projects Grant Plan project Youth learn about their community Youth ages 12–16 Partners Carry out the project Youth gain confidence in doing community work Youth engage in additional community activities This logic model illustrates the forward and backward connections (feedback loops) that are common in programs. Another chain of outcomes could be developed for the adults Time Evaluate how they did Adults Example from University of Wisconsin-Extension, Program Development and Evaluation 15

If-Then Relationships Underlying a Logic Model is a series of ‘if-then’ relationships if-then if-then if-then if-then if-then 16

How Will Activities Lead to Desired Outcomes How Will Activities Lead to Desired Outcomes? A series of if-then relationships Tutoring Program Example If Then If Then If Then If Then If Then We invest time and money We can provide tutoring 3 hrs/week for 1 school year to 50 children Students struggling academical-ly can be tutored They will learn and improve their skills They will get better grades They will move to next grade level on time Example from University of Wisconsin-Extension, Program Development and Evaluation 17

But Wait, There’s More… INPUTS OUTPUTS OUTCOMES So What? Activities Participation Short Medium Program investments Long-term What We Invest What We Do Who We Reach What results So What? What is the VALUE? Adapted from University of Wisconsin-Extension, Program Development and Evaluation 18

Developing a System (1.6) The 5 Ps of Evaluation Personnel Policies Places Programs Participant Outcomes ****These are NOT discrete—they overlap!

Things to Think about When Developing a System… Keep in mind the time and financial constraints Evals have political overtones Establish appropriate reasons for evaluation (staff want info, funder requires it, needed for decisions about continuing a program)

Purposeful Planning for Evaluation Schedule time on your calendar Involve planning board, committees, colleagues, staff, volunteers, etc. Identify what you hope to achieve (desired outcomes) Identify your goals and objectives Identify the methods you will use to measure whether or not your program met your objectives

Personnel Evaluations (Performance Appraisal or Performance Eval) Can be formative or summative Based on job description and/or plan of work Staff should receive feedback from eval Used for decisions like compensation, promotion, training, employee development Should be at least annual but not just a single activity—should have informal evals as well Evaluate performance, not personality, to help employee improve Describe how you would envision the process of personnel eval Assign duties Determine criteria based on the duties Gather info on how well employee performs Appraise performance Provide feedback to employee Make decisions and adjustments about performance (value of employee, promotions, further training, etc)

Policy Evaluation Administrative aspects include way agency is organized and operated Public policy and community surveys to measure support, opinions, or info Cost-benefit, cost-effectiveness evals Performance-based program budgets Economic impacts Assessments Cost benefits- compare cost to benefits but not just in $$ Cost –effectiveness easier because it’s ratio of cost to revenue generated (brea-even programs are 1:1) Performance –based budgets- breaks down work or program activities into detailed sub-units then compare

Places (Areas and Facilities) Areas evaluated on use, safety, and legal mandates Master planning (part of long-range planning) A&F evals often done by applying standards (ACA and P&R accreditation) GIS use very popular

Program Quality and Participants (1.7) INPUTS OUTPUTS OUTCOMES In its simplest form, a logic model is a graphic representation that shows the logical relationships between: The resources that go into the program – INPUTS The activities the program undertakes – OUTPUTS The changes or benefits that result – OUTCOMES 25

Social-Economic-Environmental Improvements Hierarchy of Effects Reactions Learning Actions Number and characteristics of people reached; frequency and intensity of contact Degree of satisfaction with program; level of interest; feelings toward activities, educational methods Changes in knowledge, attitudes, skills, aspirations Changes in behaviors and practices Participation Source: Bennett and Rockwell, 1995, Targeting Outcomes of Programs Many Extension staff will remember the Bennett hierarchy of the 1970’s that was so popular and widely used throughout Extension. The Bennett hierarchy is a precursor of the present day logic model. You can see the similarities in this graphic. Rockwell and Bennett have since developed a toolkit titled, Targeting Outcomes of Programs (TOP) that is available on the web at http://citnews.unl.edu/TOP/english/ See it for more information. 26

What results for individuals, families, communities… OUTCOMES What results for individuals, families, communities… SHORT Learning Changes in: Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation Behavioral intent MEDIUM Action Behavior Decision-making Policies Social Action LONG-TERM Conditions Social (Well-Being) Health Economic Civic Environmental OUTCOMES In order to facilitate/achieve outcomes, that extend along a continuum – or chain of outcomes from short to long-term or impact. For example, changes in knowledge, skills, attitudes and intent: Change in knowledge might be increased understanding of the purpose of a budget, or loan terms Change in skills might be how to develop a spending plan Change in attitude might be Change in confidence might be increased confidence to ask questions; go to a bank and seek service Change in intent might be Change in behavior Change in decision making Change in individual, family, financial institution, community conditions Unit of analysis?? Sebstad provides illustrative outcomes for 5 thematic areas Outcomes are the changes or benefits for individuals, families, groups, businesses, organizations, and communities. Outcomes occur along a path from short-term achievements to longer-term end results (impacts). Outcomes include Short-term: Changes in awareness, knowledge, skills, attitudes, opinions, motivation, intent such as Increased knowledge of poverty’s impact on individuals and the community Goal represents a general, big picture statement of desired results. Increased skills in leading a group Greater intention to exercise Medium-term: Changes in behaviors, decision making, action Participating youth use a spending plan Producers make informed decisions concerning farm transfer Community installs bike paths Long-term: Changes in social, economic, civic, environmental conditions such as Reduced debt Improved water quality Increased community safety The ultimate result of a program is usually referred to as “impact”. Impacts might be achieved in one year or take 10 or more years to be achieved. Such long-term impacts may or may not be reflected in the logic model, depending on scope of the initiative, purpose and audience of the logic model. C H A I N OF O U T C O M E S Example from University of Wisconsin-Extension, Program Development and Evaluation 27

Logic Model Helps with Participant-Outcomes Evaluation Provides the program description that guides evaluation process Helps match evaluation to the program Helps know what and when to measure Are you interested in process and/or outcomes? Helps focus on key, important information What do we really need to know? Adapted from University of Wisconsin-Extension, Program Development and Evaluation 28

“Outcome” (People) or “Output” (Program)—BOTH ARE IMPORTANT Outcome = tangible results of your program # of participants; # who were trained; Responses to your program (including satisfaction ratings) Output = impacts (hopefully +) on program participants Learning (knowledge, understanding, perceptions, attitudes) Skills (capabilities, specific behaviors) Conditions (increased security, stability, pride, etc.)

Find the Outcome(s) Everyone in your beginning swimming class became comfortable with putting their faces in the water. 80% of youth participants said that they enjoyed the ropes course. 8 out of 10 kids in the after-school soccer program demonstrated the correct way to pass a soccer ball. A mom called you to say that her son follows instructions better at home after being involved in your program.

Find the Outcome(s) - Answers Everyone in your beginning swimming class became comfortable with putting their faces in the water. (not an outcome) 80% of youth participants said that they enjoyed the ropes course. 8 out of 10 kids in the after-school soccer program demonstrated the correct way to pass a soccer ball. A mom called you to say that her son follows instructions better at home after being involved in your program.

Steps for Targeting Outcomes Mission Goals Outcomes Teach Staff to Implement Explore readiness to evaluate Consider how you will use your results Identify mission, goals, and objectives Learn to use a Logic Model to focus on change Complete a Logic Model Consider your outcome results Share outcome results with others Now want to think specifically about your program If not, stay here, but keep these notes for next year…you have a starting point ACA has a resource for you

Issues in Doing Research or Evaluation (1.11) Ethical Political Legal Moral

Broad Ethical Considerations Avoid bias (settings, questions, populations) Consider question content Protect confidentiality (if appropriate) Manage data/records Report results of evaluation Respect/report different opinions

“Doing the Right Thing” Political Issues Supports/refutes views and values Personal contacts Value-laden definitions Controversial recommendations Pressures to produce certain findings Know the organization’s position, don’t go beyond the data in your conclusions and recommendations, have clear purpose to eval

“Doing the Right Thing”- contd. Legal Issues Not many legal concerns except around illegal behaviors (underage drinking, drug use, illegal gambling, income tax evasion) Ethical Issues Be open with people Don’t promise more than you can do Protect the rights and privacy of your subjects Guard against coercion Get written consent & board approval Guard against harm Let participants know the results

“Doing the Right Thing”- contd. Moral Unintentional mistakes made by bias or carelessness Cultural & procedural biases Letting bias, prejudice, friendships influence outcomes Dealing with negative findings Taking too long to get the results out Being prepared to recognize the possibility of statistical errors and knowing how to explain them To guard against political, ethical, legal, moral issues: Have humility Be patient Plan Focus on practical questions, feasible issues Adopt a self-evaluation orientation- evaluate yourself! What are some dilemmas that you might face as an evaluator?

“Doing the Right Thing” for Institutional Review Boards (IRB) and “Just Because It is the Right Thing to Do” Voluntary Participation No Harm Anonymity and Confidentiality Issues of Deception Analysis and Reporting

Anonymity A researcher/evaluator cannot identify a given response with a given respondent. Confidentiality Researcher/evaluator can identify a given person's responses but promises not to do so publicly.

Informed Consent Participants in a study (research or evaluation) must base their voluntary participation on an understanding of the possible risks involved.

Institutional Review Board (Required for research—not usually for evaluation projects unless they are going to be used outside the agency) Guidelines: Minimize the risk to subjects Risk must be reasonable to benefits of the study Informed consent Adequate detail should be provided about the study in lay language Privacy must be maintained Freedom of choice, free to withdraw at any time