EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

Agenda Quasi-evaluation studies Activity (if time allows)

Quasi-evaluation studies

Address specific questions (often employing a wide range of methods) Advocate use a particular method Whether the questions or methods are appropriate for assessing merit and worth is a secondary consideration Both are narrow in scope and often deliver less than a full assessment of merit and worth

Approach 7: Objectives-based studies Advance organizers – Statement of program objectives Purposes – To determine to what extent a program achieved objectives Sources of questions – Objectives as defined by staff, funder, or evaluator Questions – To what extent were each of the stated objectives met?

Objectives-based evaluation results from a national research center

Methods – Any relevant method for determining to which operationally defined objectives were met Pioneers – Ralph Tyler Use considerations – Must have clear, supportable objectives Strengths – Ease of application Weaknesses – Narrowness and inability to identify positive and negative side effects

Approach 7: The success case method Advance organizers – Comparison between successful and unsuccessful instances Purposes – To determine how well and in what respects a program is ‘working’ Sources of questions – Generally from program providers Questions – What are the noteworthy successes? – How were successes produced? – What factors contributed to success/failure?

Methods 1.Focus and plan the study 2.Create an impact model 3.Survey all participants 4.Interview a sample of success and nonsuccess cases 5.Communicate findings, conclusions, and recommendations Pioneers – Robert Brinkerhoff Use considerations – Intended to assist service providers in increasing ‘successes’ and decreasing ‘nonsuccesses’ Strengths – Ease of application – Use for improvement Weaknesses – Narrowness of scope

Approach 9: Outcome evaluation as value-added assessment Advance organizers – System-wide measures of ‘growth’ or ‘gains’ Purposes – ‘Value added’ by a program and its constituent parts Sources of questions – Oversight bodies Questions – What parts of a program contribute most to ‘growth’ or ‘gains’?

Methods – Gain score analysis, hierarchical linear modeling, etc. Pioneers – Raudenbush, Sanders, Horn, Timms, etc. Use considerations – Can be used to make and/or support policy decisions Strengths – Longitudinal rather than cross-sectional Weaknesses – Potential misuse by policy makers in placing ‘blame’

Approach 10: Experimental and quasi-experimental studies Advance organizers – Cause-and-effect hypotheses, competing treatments, etc. Purposes – To determine causal relationships between independent and dependent variables Sources of questions – Researchers, developers, policy makers, etc. Questions – To what extent is one treatment superior to another?

Methods – Random or other method of assignment to conditions Pioneers – Campbell, Cook, Shadish Use considerations – Addresses only one particular type of question Strengths – Strong causal conclusions (if assumptions are met) Weaknesses – Requires substantial expertise, time, money, etc.

Approach 11: Cost studies Advance organizers – Costs associated with program inputs, outputs, and outcomes Purposes – The costs and outcomes of one more more alternatives Sources of questions – Policy makers, planners, taxpayers, etc. Questions – What are the costs of obtaining desired outcomes?

Methods – Analysis of monetary and nonmonetary units Pioneers – Levin, McEwin, Yates, etc. Use considerations – Expertise required Strengths – ‘Bottom line’ conclusions of interest to most decision makers Weaknesses – Often difficult to validly execute

Approach 12: Connoisseurship and criticism Advance organizers – Specialized expertise, sensitivities, tacit knowledge, etc. Purposes – To describe, appraise, and illuminate Sources of questions – Determined by the ‘connoisseurs’ or ‘critics’ Questions – What are a program’s salient characteristics, strengths, and weaknesses?

Methods – Perceptual sensitivity, prior experience, refined insights, etc. Pioneers – Elliot Eisner Use considerations – An audience willing to accept the approach Strengths – Exploitation of refined expertise Weaknesses – Objectivity and reliability

Approach 13: Theory-based evaluation Advance organizers – A carefully specified ‘theory’ of how a program is intended to operate Purposes – To determine the extent to which a program is ‘theoretically sound’ Sources of questions – Determined by the guiding program theory Questions – To what extent does the program theory ‘work’ or not?

Methods – Any method appropriate for testing the program theory Pioneers – Chen, Donaldson, Weiss, Rogers, Rossi, etc. Use considerations – Difficulty in applying the approach Strengths – Useful for determining potential ‘measurement’ variables Weaknesses – Few programs are grounded by validated ‘theories’

Approach 14: Meta-analysis Advance organizers – Sufficient studies of the same or similar programs Purposes – To assemble and (statistically) integrate findings from multiple studies of the same or similar programs Sources of questions – Policy makers, ‘research repositories’, etc. Questions – What is the average effect of a particular type of program?

Methods – Statistical methods for integrating study results (varies widely) Pioneers – Glass Use considerations – Major source of contemporary ‘best practices’ across a variety of domains Strengths – Evidence of effectiveness over units, treatments, observations, and settings Weaknesses – Exclusive emphasis on program outcomes

Activity

We will split the class into two sections (1 and 2) – In each section, appoint one student to chair your appointed group – Each member of section 1 should select one of the approaches discussed today and discuss why it is useful – Members of section 2 should listen and take notes – Members of section 2 should then outline weaknesses of the selected approaches – Finally, the chair of each group should discuss the potential strengths, weaknesses, and utility of the selected approaches

Encyclopedia Entries Bias Causation Checklists Chelimsky, Eleanor Conflict of Interest Countenance Model of Evaluation Critical Theory Evaluation Effectiveness Efficiency Empiricism Independence Evaluability Assessment Evaluation Use Fournier, Deborah Positivism Relativism Responsive evaluation Stake, Robert Thick Description Utilization of Evaluation Weiss, Carol Wholey, Joseph