Program Evaluation

Lecture Overview  Program evaluation and program development  Logic of program evaluation (Program theory)  Four-Step Model  Comprehensive Quality Programming  Empowerment Evaluation  Group Activity

Program Development  refers to the planning and implementation of a program determining its purposes and setting goals and outcomes, evaluating program process, and evaluating attainment of desired program outcomes and wider impacts.

History of Program Evaluation  1930’s human service field grew (post depression)  WWII programs were launched to meet the needs for urban development and housing, technological and cultural education, occupation training, and preventive health activities.  By the end of the 1950’s program evaluation was commonplace  1960’s increased papers on evaluation research  1970’s, evaluation was a distinct field  1980’s benefits and costs of programs  1990’s new social problems and booming economy

Program Evaluation  Refers to the collection of evidence to judge the effectiveness of a community programs or other community interventions (com psych definition)  The use of social research procedures to systematically investigate the effectiveness of social intervention programs (research definition).

Project Drug Abuse Resistance Education D.A.R.E.  Most popular school based intervention  Little evidence on its effectiveness  Limited positive influence on psychological factors (self-esteem)  How can this lack of measurable outcomes in a popular intervention be explained?

Reasons Why Programs May Be Unsuccessful  Theory Failure: concerns program theory which is the rationale for why a particular intervention is considered appropriate for a particular problem with a specific target population  Implementation Failure: concerns quality of program implementation (ie: program may not be generalizable)

Effects do not occur when…….  The underlying assumptions of program theory are incorrect.  The program, even if implemented well, doesn’t affect the variables specified by program theory.  Or the activity or program is not implemented adequately

Logic Model  Graphic representation of the program.  Shows the logical connections between the conditions that contribute to the need for a program in a community, the activities aimed at addressing these conditions, and the outcomes and impacts expected to result from the activities.

Evaluation “Terminology”  Stakeholders  Fiduciary  Funding sources  Need Assessment  Evaluation Instruments

Four Step Model of Program Evaluation (Lindsay & Wandersman, 1991)  Step 1: Identify Goals and Desired Outcomes  Step 2: Process Evaluation  Step 3 Outcome Evaluation  Step 4: Impact Evaluation:

STEP 1) Identifying Goals and Desired Outcomes  What is your project striving for?  Goals set a framework for specific outcomes. Program Developers MUST describe:  The primary goals of the program  Target population  Outcomes desired

STEP 2) Process Evaluation  Monitoring program activities helps organize program efforts  Provides information to help manage the program accountability that the program is conducting what was done and who was reached  After an evaluation of outcomes and impacts, the evaluation can provide information about why the program worked or did not work  Can help decide whether or not you are ready to assess the effects of your program  What were the intended and actual activities of the program/ Did you learn anything.

STEP 3) Outcome/Impact Evaluation  Assess the immediate effects of a program “the bottom line”  Looks at the desired outcomes defined in Step 1 and seeks evidence regarding the extent to which those outcomes were achieved  Should be closely linked to goals, but more specific

STEP 4) Impact Evaluation  Concerned with the ultimate effects of the program  Impacts are ultimate or longer-term effects of the program

Comprehensive Quality Programming: 9 Strategies  Why is the intervention or program needed?  What are the program’s goals, target populations and desired outcomes?  How does your program use the scientific knowledge and best practice of what works?  How will this new program fit in with existing programs  How will you carry out the program  How well was the program carried out  How did the program work?  What can you do to improve the program the next time you do it?  If the program or parts of the program were effective, what are you doing to continue institutionalize the program

Accountability Questions & Corresponding CQP Strategies  Why is the intervention or program needed?  What are the program’s goals, target populations and desired outcomes?  How does your program use the scientific knowledge and best practice of what works?  How will this new program fit in with existing programs  How will you carry out the program  How well was the program carried out  How did the program work?  What can you do to improve the program the next time you do it?  If the program or parts of the program were effective, what are you doing to continue institutionalize the program Needs and assets assessments Identifying goals and desired outcomes Review research literature and best practices Community feedback and planning Planning, implementation Process evaluation Outcome and impact evaluations Continuous program improvement Institutionalization strategies

Empowerment Evaluation  Program planners and developers learn the basics of program evaluation in ongoing program improvement.  Help program developers and staff to achieve their program goals, by providing the with the tools for assessing and improving the planning, implementation, and results of their program.

Limitations of Evaluation  Use of an outside evaluator  Evaluation feedback is provided at the end of the program  Evaluation findings are perceived as being too complex

Conclusions  Program evaluation is necessary to find out whether a program is working  Evaluation can be used for program planning and program implementation  As society becomes more concerned about accountability and results, evaluation will become more necessary.

Group Activity  Apply 4 step model in terms of your group project.  Answer CQP questions