Program Evaluation, Archival Research, and Meta-Analytic Designs Chapter 12 Program Evaluation, Archival Research, and Meta-Analytic Designs

Program evaluation Program evaluation A kind of applied research which has the goal of determining if a policy or specific program is doing what its creators intended it to do. Not a research design, rather it indicates something about the motivation of the research. It is applied research with a focus on evaluation. Program evaluators also determine if the program is truly needed, is cost effective, is being run efficiently, and they will offer suggestions for improvement.

Program evaluation Determining need Needs analysis – a way of determining whether there are enough people who would benefit from and use the program and whether the program would meet the needs of those people. Determining need might involve examining available data, surveying potential users of the program, and/or conducting focus groups within the target community.

Program evaluation Selecting outcome measures Once it has been determined that a need exists, that the program can be expected to meet that need, and that people will use the program, program evaluators must make decisions about what kinds of measures should be taken so that the program can be properly evaluated. Care must be taken to match the outcome measures with the goals of the program and to minimize any bias that might be introduced by the people doing the measuring.

Program evaluation Ethical constraints on program evaluation research There are limitations on the kind of research design that can be ethically used in program evaluation. Program evaluation researchers tend to use designs that might lack some experimental control but are ethically more acceptable (e.g.. non-equivalent control group, time series, and pretest-posttest designs).

Archival research Archival information – information that has been collected for some reason other than the research project at hand. Archival research – conducted by researchers using archival information. Archive – refers to both the information itself and where it is housed. Strengths of archival research: The information base is huge and extensive. No problems with participant reactivity. Potential problem: experimenter bias  researchers must take steps to reduce the probability of this kind of bias.

Meta-analysis Statistical technique that allows us to combine the effect sizes reported in experimental and correlational studies of the same underlying variables. The variables may be different in terms of their operational definitions but the underlying construct is considered the same. Allows us to better estimate the true relationship among variables than each study does alone.