EVALUATION AND RESEARCH © LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON

STRUCTURE OF THE CHAPTER Similarities between research and evaluation Differences between research and evaluation Connections between evaluation, research, politics and policy making © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

DEFINING EVALUATION The provision of information about specified issues upon which judgements are based and from which decisions for action are taken. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

COMPARING RESEARCH AND EVALUATION Origins: Research questions originate from scholars working in a field; evaluation questions issue from stakeholders. Audiences: Evaluations are often commissioned and they become the property of the sponsors and are not for the public domain; research is disseminated widely and publicly. Purposes: Research contributes to knowledge in the field, regardless of its practical application, and provides empirical information, i.e. ‘what is’; evaluation is designed to use that information and those facts to judge the worth, merit, value, efficacy, impact and effectiveness of something, i.e. what is valuable. Research is conducted to gain, expand and extend knowledge; evaluation is conducted to assess performance and to provide feedback. Research is to generate theory; evaluation is to inform policy making. Research is to discover; evaluation is to uncover. Research seeks to predict what will happen; evaluation concerns what has happened or what is happening. Stance: The evaluator is reactive (e.g. to a programme); the researcher is active and proactive. Status: Evaluation is a means to an end; research is an end in itself. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

COMPARING RESEARCH AND EVALUATION Focus: Evaluation is concerned with how well something works; research is concerned with how something works. Outcome focus: Evaluation is concerned with the achievement of intended outcomes; research may not prescribe or know its intended outcomes in advance (science concerns the unknown). Participants: Evaluation focuses almost exclusively on stakeholders; research has no such focus. Scope: Evaluations are concerned with the particular, e.g. a focus only on specific programmes. They seek to ensure internal validity and often have a more limited scope than research. Research often seeks to generalize (external validity) and, indeed, may not include evaluation. Setting of the agenda: The evaluator works within a given brief; the researcher has greater control over what will be researched (though often constrained by funding providers). Evaluators work within a set of ‘givens’, e.g. programme, field, participants, terms of reference and agenda, variables; researchers create and construct the field. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

COMPARING RESEARCH AND EVALUATION Relevance: Relevance to the programme or what is being evaluated is a prime feature of evaluations; relevance for researchers has wider boundaries (e.g. to generalize to a wider community). Research may be prompted by interest rather than relevance. For the evaluator, relevance must take account of timeliness and particularity. Timeframes: Evaluation begins at the start of the project and finishes at its end; research is ongoing and less time-bound (though this may not be the case with funded research). Uses of results: Evaluation is designed to improve; research is designed to demonstrate or prove. Evaluation informs decision making; research provides a basis for drawing conclusions. Evaluations might be used to increase or withhold resources or to change practice; research provides information on which others might or might not act, i.e. it does not prescribe. Decision making: Evaluation is used for micro decision making; research is used for macro decision making. Data sources and types: Evaluation has a wide field of coverage (e.g. costs, benefits, feasibility, justifiability, needs, value for money), so evaluators employ a wider and more eclectic range of evidence from an array of disciplines and sources than researchers. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

COMPARING RESEARCH AND EVALUATION Ownership of data: The evaluator often cedes ownership to the sponsor, upon completion; the researcher holds onto the intellectual property. Politics of the situation: The evaluator may be unable to stand outside the politics of the purposes and uses of, or participants in, an evaluation; the researcher provides information for others to use. Use of theory: Researchers base their studies in social science theory; this is not a necessary component of evaluation. Research is theory-dependent; evaluation is ‘field-dependent’, i.e. not theory-driven but derived from the participants, the project and stakeholders. Researchers create the research findings; evaluators may (or may not) use research findings. Reporting: Evaluators report to stakeholders/commissioners of research; researchers may include these and may also report more widely, e.g. in publications. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

COMPARING RESEARCH AND EVALUATION Standards for judging quality: Judgments of research quality are made by peers; judgements of evaluation are made by stakeholders. For researchers, standards for judging quality include validity, reliability, accuracy, causality, generalizability, rigour; for evaluators, to these are added utility, feasibility, involvement of stakeholders, side-effects, efficacy, fitness for purpose (though, increasingly, utility value and impact are seen as elements for judging research). © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

SIMILARITIES BETWEEN EVALUATION AND RESEARCH Evaluation can examine the effectiveness of a program or policies, as can research. Evaluation and research share the same methodologies (styles, instrumentation, sampling, ethics, reliability, validity, data analysis techniques, reporting and dissemination mechanisms). © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

DIFFERENCES BETWEEN EVALUATION AND RESEARCH (Smith, M. & Glass, G DIFFERENCES BETWEEN EVALUATION AND RESEARCH (Smith, M. & Glass, G. (1987) Research and Evaluation in the Social Sciences. New Jersey: Prentice Hall) The intents and purposes of the investigation The scope of the investigation Values in the investigation The origins of the study The uses of the study The timeliness of the study Criteria for judging the study The agendas of the study © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

DIFFERENCES BETWEEN EVALUATION AND RESEARCH (Norris, N DIFFERENCES BETWEEN EVALUATION AND RESEARCH (Norris, N. (1990) Understanding Educational Evaluation. London: Kogan Page) The motivation of the enquirer The objectives of the research Laws versus description The role of explanation The autonomy of the enquiry © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

DIFFERENCES BETWEEN EVALUATION AND RESEARCH (Norris, N DIFFERENCES BETWEEN EVALUATION AND RESEARCH (Norris, N. (1990) Understanding Educational Evaluation. London: Kogan Page) Properties of the phenomena that are assessed Universality of the phenomena studied Salience of the value question Investigative techniques Criteria for assessing the activity Disciplinary base © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

CONFORMATIVE EVALUATION (Stronach, I. and Morris, B CONFORMATIVE EVALUATION (Stronach, I. and Morris, B. (1994) Polemical notes on educational evaluation in an age of ‘policy hysteria’. Evaluation and Research in Education, 8 (1-2), pp. 5-19) Short term, takes project goals as given. Ignores the evaluation of longer-term outcomes. Gives undue weight to the perceptions of programme participants who are responsible for the successful development and implementation of the programme: ‘over-reports’ change. Neglects/‘under-reports’ the views of some practitioners and critics. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

CONFORMATIVE EVALUATION Adopts an atheoretical approach, and regards the aggregation of opinion as the determination of significance. Involves a tight contractual relationship with programme sponsors that disbars public reporting or encourages self-censorship to protect future funding. Risks implicit advocacy of the programme in its reporting style. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

MODELS OF EVALUATION Survey: cross-sectional, longitudinal Experiment Illuminative The CIPP (Stufflebeam) and the Countenance model (Stake) Context, Input, Process, Product; Antecedents, Transactions, Outcomes; Look for congruence between what was intended to happen and what actually happened in these four areas. Objectives How far have the objectives been achieved. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

STAKE’S MODEL OF EVALUATION Congruence between intentions & observations – what actually happened   INTENTIONS Congruence OBSERVATIONS Intended antecedent         Actual antecedents Intended transactions Actual transactions Intended outcomes Actual outcomes Antecedents = initial conditions Transactions = processes, what takes place during the programme © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

RESEARCH, POLITICS AND POLICY MAKING Politics, research and evaluation are inextricably linked in respect of: Funding Policy-related research Commissioned research Control and release of data and findings Dissemination of research How does research influence policy? Who judges research utilization? Consonance with political agendas © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

RESEARCH, POLITICS AND POLICY MAKING Researchers and policy makers may have conflicting: Interests Agendas Audiences Time scales Terminology Concern for topicality © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

RESEARCH, POLITICS AND POLICY MAKING Policy makers like: Simple impact model Superficial facts Unequivocal data Short-term solutions Simply, clear remedies for social problems Certainty Positivist methodologies Researchers work with: Complex models Complex data Uncertain findings Longer-term time scales Subtle, provisional data on complex issues Conjecture Diverse methodologies © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors