A brief overview What is program evaluation? How is an evaluation conducted? When should it be used? When can it be used? Used with Permission of: John R. Slate

Presentation Outline n Definitions n Purposes n Types n Key concepts of evaluative research n Research designs n Requirements for program evaluation

A Definition of “Program” “An organized set of resources and activities directed toward a common purpose or goal”

Two Definitions of “Program Evaluation” n “... an assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives.” ( Source - Government Performance and Results Act (GPRA)) n “The application of scientific research methods to assess program concepts, implementation and effectiveness” ( Source - General Accounting Office, Designing Evaluations, Report GAO/PEMD )

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction 5

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it, set targets and program objectives 6

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it, set targets and program objectives Create program infrastructure & management, administrative & information systems, develop performance measures 7

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it, set targets and program objectives Create program infrastructure & management, administrative & information systems, develop performance measures Determine required levels of human resources and material support 8

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it, set targets and program objectives Create program infrastructure & management, administrative & information systems, develop performance measures Determine required levels of human resources and material support Implement action plan (Program operation & management, performance measurement, corrective action, etc...,.) 9

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it, set targets and program objectives Create program infrastructure & management, administrative & information systems, develop performance measures Determine required levels of human resources and material support Implement action plan (Program operation & management, performance measurement, corrective action, etc...,.) Program effectiveness, impact, efficiency evaluations to determine the continue need for the program, alter program design, resource requirements, etc.. 10

Types of Program Evaluations n Formative n Formative - Judging the worth of a program while activities are forming or in process focus is on the process more than the outcome can help make in- process improvements often involves a small scale field test 11

Types of Program Evaluations n Formative n Formative - Judging the worth of a program while activities are forming or in process focus is on the process more than the outcome can help make in- process improvements often involves a small scale field test n Summative n Summative - Judging the effectiveness of a fully operating or completed program focus in on outcome and overall program worth can help decisions to expand, terminate, modify usually encompasses the entire program 12

Questions Asked by Program Evaluations* n Descriptive - Statistics on inputs, outputs, and outcomes; how does the program work? n Normative - What is the expected performance (goal) of the program in relation to actual achievement? n Impact - If a goal is not met, why not? Program evaluations must establish a cause/effect relationship between an unmet goal and program activities or other, external factors * from Designing Evaluations, GAO/PEMD

Decisions Program Evaluation Can Help Make n Continue or discontinue a program n Improve policies and procedures n Add or drop specific program elements n Institute similar programs elsewhere n Allocate resources n Accept or reject approaches & theories

Key concepts of Evaluative Research n empirical - based on valid, reliable data n replicable - study can be repeated in exactly the same way in another time, place or setting n falsifiable - hypothetical cause/effect relationships can be demonstrated or not According to standards in the research and academic communities, program evaluations should strive for scientific proof and method, i.e.., they should be:

Evaluation Methodology (highly simplified) n Define program n Identify outcome goals and objectives n Model hypothetical cause/effect relationships between program activities and outcomes n Develop goal/objective measurement criteria and desired achievement levels n Locate, collect and analyze data on program participants (and maybe control group) n Compare actual results with target levels

Evaluation Strategies and Designs* * Based on Designing Evaluations, GAO/PEMD

Evaluation Strategies and Designs* * Based on Designing Evaluations, GAO/PEMD

Evaluation Strategies and Designs* * Based on Designing Evaluations, GAO/PEMD

Evaluation Strategies and Designs* * Based on Designing Evaluations, GAO/PEMD

Evaluability Assessment n A study to determine if, when, and how a program can be evaluated n Prevents premature impact and outcome evaluations n Usually involves: clarification of the intent of the program formulation of “testable” cause/effect statements determines what can be measured and how assesses validity, reliability, and relevance of data on which the evaluation would based

Circumstances When Program Evaluation is Not a Good Idea n When there is no question about the program n When there is no clear program structure or focus n When program activities cannot be distinguished from other activities n When people cannot agree on what the program is trying to achieve n When valid measurement methods and data do not exist and cannot be created n When the study would serve no useful purpose

Overview Program Evaluation Prerequisites n An operating “program”, i.e.., a distinct set of activities and resources with a common purpose and focus n Agreement on program goals and objectives n Agreement on program metrics for its goals and objectives n Agreement on what constitutes program “success” n Existence, or ability to develop, valid data n Absence of legal, administrative, or cultural barriers to the study n Agreement on intended use of the study