Semester 2: Lecture 9 Analyzing Qualitative Data: Evaluation Research Prepared by: Dr. Lloyd Waller ©

Evaluation Research Interventions are introduced to institutions, groups, organizations and communities usually with the goal of achieving certain goals and objectives or at least with the expectations for certain outcomes. Interventions often need evaluation – mid-term evaluation, or Quarterly evaluation or end of programme evaluations - to determine whether or not it has achieved its goals. Evaluations are normally conducted by doing Evaluation Research Evaluation Research is a type of Research Methodology where research is undertaken or the purpose of determining the impact of intervention.

Evaluation Research Political Science Research and Evaluation Research closely related to, but distinguishable from more traditional social research. utilizes many of the same methodologies used in traditional social research Requires special skills not usually taught in political science research: –group skills –management –political dexterity –sensitivity to multiple stakeholders

Evaluation Research The Goals of Evaluation Research to provide "useful feedback“ –sponsors –Donors –client-groups –Administrators –staff, and –other relevant constituencies to aid in decision-making to aid in policy-making to improve an intervention to promote change

Evaluation Research Evaluation Strategies Scientific-Experimental Models –Influenced by the social sciences - they prioritize on the desirability of impartiality, accuracy, objectivity and the validity of the information generated –Follows guidelines which are presumed by the research community –Includes techniques such as: experimental research designs quasi-experimental research designs cost-effectiveness cost-benefit analysis; –Emphasis is on causal connections, emphasizing measurement as an integral part of evaluation. –Quantifiable measures are critical to this type of evaluation, such as: Psychometrics Econometrics

Evaluation Research Evaluation Strategies Management-Oriented Systems Models –Primary emphasis is on comprehensive process management, with ongoing evaluation by management a key element –Strategies include: MBO (Management by Objectives) PERT (Program Evaluation and Review) CPM (Critical Path Method) CIPP (Context, Input, Process, Product) Information Auditing Information Mapping Logical Framework

Evaluation Research Evaluation Strategies Participant-oriented –Emphasize the importance of the participants in the process, especially the clients and users of the program or technology –Focus on the claims, concerns, and issues of a variety of potential stakeholders: agents (implementers, funders) beneficiaries (included target groups) victims (excluded target groups)

Evaluation Research Evaluation Strategies Qualitative/anthropological Models –These include the various naturalistic forms of evaluation, emphasizing the value of subjective human interpretation of the observations made. –Evaluator's role is to provide expert judgments on the outcomes of interest. –Strategies include: critical theory grounded theory'

Evaluation Research Types of Evaluation There are many different types of evaluations depending on the object being evaluated and the purpose of the evaluation –Formative evaluation –Summative evaluation.

Evaluation Research Types of Evaluation Formative evaluation –needs assessment determines who needs the program, how great the need is, and what might work to meet the need –evaluability assessment determines whether an evaluation is feasible and how stakeholders can help shape its usefulness –structured conceptualization helps stakeholders define the program or technology, the target population, and the possible outcomes –implementation evaluation monitors the fidelity of the program or technology delivery –process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures.

Evaluation Research Types of Evaluation Summative evaluation. –outcome evaluations investigate whether the program or technology caused demonstrable effects on specifically defined target outcomes –impact evaluation is broader and assesses the overall or net effects -- intended or unintended -- of the program or technology as a whole –cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in terms of their dollar costs and values –secondary analysis reexamines existing data to address new questions or use methods not previously employed –meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgement on an evaluation question

Evaluation Research Types of Evaluation Evaluation Questions and Methods –Formative research - major questions and methodologies What is the definition and scope of the problem or issue, or what's the question? Formulating and conceptualizing methods might be used including brainstorming, focus groups, nominal group techniques, Delphi methods, brainwriting, stakeholder analysis, synectics, lateral thinking, input-output analysis, and concept mapping. Where is the problem and how big or serious is it? The most common method used here is "needs assessment" which can include: analysis of existing data sources, and the use of sample surveys, interviews of constituent populations, qualitative research, expert testimony, and focus groups.

Evaluation Research Types of Evaluation Evaluation Questions and Methods –Formative research - major questions and methodologies How should the program or technology be delivered to address the problem? Some of the methods already listed apply here, as do detailing methodologies like simulation techniques, or multivariate methods like multiattribute utility theory or exploratory causal modeling; decision-making methods; and project planning and implementation methods like flow charting, PERT/CPM, and project scheduling. How well is the program or technology delivered? Qualitative and quantitative monitoring techniques, the use of management information systems, and implementation assessment would be appropriate methodologies here.

Evaluation Research Types of Evaluation Evaluation Questions and Methods –Summative Evaluation - major questions and methodologies What type of evaluation is feasible? Evaluability assessment can be used here, as well as standard approaches for selecting an appropriate evaluation design. What was the effectiveness of the program or technology? One would choose from observational and correlational methods for demonstrating whether desired effects occurred, and quasi- experimental and experimental designs for determining whether observed effects can reasonably be attributed to the intervention and not to other sources. What is the net impact of the program? Econometric methods for assessing cost effectiveness and cost/benefits would apply here, along with qualitative methods that enable us to summarize the full range of intended and unintended impacts.

Evaluation Research Types of Evaluation Evaluation Questions and Methods –Summative Evaluation - major questions and methodologies What type of evaluation is feasible? Evaluability assessment can be used here, as well as standard approaches for selecting an appropriate evaluation design. What was the effectiveness of the program or technology? One would choose from observational and correlational methods for demonstrating whether desired effects occurred, and quasi- experimental and experimental designs for determining whether observed effects can reasonably be attributed to the intervention and not to other sources. What is the net impact of the program? Econometric methods for assessing cost effectiveness and cost/benefits would apply here, along with qualitative methods that enable us to summarize the full range of intended and unintended impacts.

Input Measurement Output/Performance Measurement Impact/Outcomes Assessment Evaluation Research

Steps in Evaluation Research WHAT ARE THE PURPOSE AND GOALS OF THE PROGRAM? WHAT EVALUATION QUESTIONS DO I NEED TO ANSWER WHAT TYPE OF DESIGN WILL GIVE ME THE DATA I NEED (Experimental, Quasi-Experimental) WHAT TOOLS WILL GIVE ME THE DATA I NEED (Methods of Data Collection) HOW DO I COLLECT THE DATA (Procedure) HOW DO I ANALYZE THE DATA AND MAKE THE FINDINGS USEFUL