Unit 2: Asking Questions. Getting Answers Program Evaluation Dr. David Thomason Kaplan University

Getting to The Information: Asking the Right and Sometimes Wrong Questions! Many opportunities exist for in-depth interviews and qualitative evaluation. Let’s brainstorm about occasions or situations where you might wish to obtain the insider’s perspective on what it is like to be a participant in a given program or agency. For instance, interviews might be conducted with residents of a Salvation Army shelter or of the street youth in a shelter for runaways. Where else?

Researcher Bias: Neutrality? A problem with any type of data collection is that bias may enter from the way an instrument is constructed, the researcher’s values or interpretation, or even in the selection of respondents. How might you identify biases that could present. While it may be easier to identify the biases of others, consider what biases might creep into your own interpretation of data. For example, think about their interviewing specific populations: men in an alcohol detox program, women in a shelter for domestic violence, men in a treatment program for anger management, women in prison without the possibility of parole.

Informing the Audience In qualitative evaluation the evaluator is more involved on a personal level with the participants of a program than a quantitative evaluator would be. Think through the issue of informed consent. Do you need to obtain it? If so, how do you secure consent? Would the findings be different than if the evaluators simply observed clients in a public program? How much should the evaluator tell the participants about the study?

Hawthorne Effect What is it? Does it tell us something about observing activity and the role of anonymity or removal of the researcher?

Hypothetical Evaluation? Can you think of a hypothetical evaluation? Let’s develop an interview guide for an evaluation project We will then critique it from the perspective of ways to probe deeper, to obtain contrasts, and constructing ways to verify the responses.

American Evaluation Association The website for the American Evaluation Association ( succinctly outlines the various models of program evaluation in great detail noting that often some models are used together in one program evaluation. Be aware of the importance of correctly framing the question(s) that the evaluation is to answer before any evaluation model is used. In addition, the website outlines and explains the following five program evaluation principles: (1) systematic inquiry, (2) competence, (3) integrity/honesty, (4) respect for people, and (5) responsibilities for general and public welfare. Please refer to and direct the students to the website for a full explanation of these principles. These principles will be discussed again in the Unit on Ethics and Professional Judgment.