Summary Slide PROGRAM EVALUATION PA430 – Week 7 – 2/29-3/1/00

PROGRAM EVALUATION A systematic assessment of the operation and/or outcomes of a program or policy Comparison against a set of explicit of implicit standards Means of contributing to the improvement of the program or policy

Key Elements Systematic assessment Focus is operations and outcomes of program Standards for comparison Purpose is to contribute to improvement or program/policy

Definitions Program – a specific government activity (examples: Head Start, toxic waste cleanup, immunization of children) Policy – a broad, officially recognized statement of objectives tied to a variety of government activities (examples: health and safety of medical products)

Main questions of evaluation research Outcome evaluation Is the program reaching the goals it was meant to accomplish? Includes both intended and unintended results Impact evaluation What happens to participants as a result of the program?

Main questions of evaluation research Process evaluation What is the program actually doing? What kinds of services are being provided? Are clients satisfied with the program? Helps in understanding of outcome data

Reasons for Evaluation Research Formative – to help improve design of program. Improve delivery of program/activity. Summative – to provide information at the end of a program (or at least one cycle of it) about whether it should be continued or modified

How is evaluation research different from other types of research? Utility – it is intended for use by program administrators Program-derived questions – questions are derived from concerns of program communities Judgmental quality – Tends to compare the “what is” with “what should we be”

Action setting – must be field-based research Action setting – must be field-based research. Goals of evaluation sometimes conflict with goals of program or its administrators Role conflicts – it is difficult for program administrators to remove themselves from commitment to and positive view of their program

Publication – basic research is usually published in academic journals Publication – basic research is usually published in academic journals. Most evaluation research is not published – remains “in-house” Allegiance – research has dual role – both program improvement and contribution to understanding of a particular policy area or evaluation research

Who performs evaluation research? Three basic ways Assign the task to a staff member of the agency Hire an outside, independent evaluation or consulting firm (sometimes a university researcher) Open biding to all potential evaluations (often through an RFP – request for proposal) Periodic evaluation of programs often required as condition of grant (either public or private granting agency)

Inside vs Outside Evaluation Is the evaluator a member of the agency staff or not Concerns over the evaluator’s perspective Do they have a stake in the study results Are they too removed from the program (too “ivory tower”) Is the evaluator competent

Objectivity – what are the possible biases of the evaluator Program knowledge – how well does the evaluator understand the program (process, goals, etc.) Potential for utilization – evaluators often must take an active role in moving evaluation from report to action (implementation)

Step 1: Understand the Program Begin with a general knowledge of the field but quickly develop an in-depth understanding of the specific program. How? Research by others in the general and specific area Written materials of the program Field research including interviews

Why this is important To develop a sense of the issues – separate the wheat from the chaff To formulate relevant, incisive questions To interpret the evidence/findings To make sound recommendations for program change or continuation To write a thorough, useable report

Step 1: Understand the Program Develop a characterization of the program (reality vs the illusion) read previous evaluations talk to program directors observation data-based inquiry What is the program trying to achieve? Begin with official goals (if available) get other, more contemporary information from program managers communicate with clients

Step 1: Understand the Program How does the program expect to achieve its goals? Not just did the program work, but what made it work examine program’s theories of change - the set of beliefs that underlie action an explanation of the causal links between program and outcomes

Step 2: Plan the Evaluation Identify key questions for study decide on analysis method: quantitative, qualitative, or both develop measures to answer questions plan data collection to operationalize key measures plan an appropriate research design collect and analyze data write and disseminate report promote appropriate use of the results

Step 2: Plan the Evaluation Additional considerations long-term vs short-term study questions should examine both intended and unintended impacts of program practicalities (clout of stakeholders, uncertainties, decision timetable) advisory committee ethical issues

Step 3: Develop measures Desired outcomes effects on persons served effects on agencies effects on larger systems (networks) effects on the public unintended outcomes both positive and negative Interim markers of progress towards outcomes real changes desired may lie far in future

Step 3: Develop measures components of program implementation (program processes) how the program is carried out how the program operates and for whom program quality resources, inputs, and environment budgets, staff, location management, years of operation client eligibility standards

Step 4: Collect data Data sources existing data informal interviews observations formal interviews, written questionnaires program records data from outside institutions

Step 5: Select a program design Identify people/units to be studied how study units will be selected kinds of comparisons to be drawn timing of the investigation Outcome studies underlying logic is: compare program participants before and after receiving program compare participants with non-participants

Step 5: Select a program design Informal designs self-evaluation by administrators, staff, and clients expert judgment (outsider knowledge) Formal designs post-test only pre-test, post-test comparison group time series designs

Step 6: Analyze and interpret data Whether quantitative or qualitative data, goal is to convert a mass of raw data into a coherent, organized report Types of analytical strategy describe, count factor, cluster (divide into parts) compare, find commonalities covariation tell the story

Step 7: Write the report What should the report look like? It depends!! May require more than one report (different audiences) comprehensive report may be required by sponsor of agency/evaluation - may be too much for most audiences executive summary

Step 7: Write the report Possible topics (full report) summary of study results findings, implications, recommendations problem with which program deals nature of the program context (history, sponsorship, setting) beneficiaries staff how study was conducted suggestions for further evaluation

Step 7: Write the report Other report types summary report for clients and public short, focus on highlights executive summary broader, useful for variety of audiences Ultimate goal: a report that is clear, timely, generalizable to other similar programs, inclusive of the organization's views, and of high utility