EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2012

Agenda Stage Two theories – Joseph Wholey – Robert Stake Questions and discussion Encyclopedia of Evaluation entries

“Feedback information about how a program is working to improve its operation, is missing from most local and state evaluation activities” — Joseph S. Wholey

Biographical Sketch Ph.D. in Philosophy and M.A. in Mathematics, Harvard University B.A. in Mathematics, Catholic University Professor of Public Policy and Planning at the University of Southern California Senior Advisor for Evaluation Methodology at GAO Author of several influential textbooks

Wholey’s View of Evaluation Began under the auspices of truth seeking and experimental methods His later work emphasized instrumental use of evaluation Overall, his central concern was producing actionable information aimed at those who are in positions to take decisions His choice of method was quantitative in orientation

Wholey’s Influence Largely at the federal government level Likely had an influence on 1993’s Government and Performance Results Act (GPRA) and the later Program Rating Assessment Tool (PART) Better government through timely, useful information on program performance

Wholey’s Major Contributions Evaluability assessment Performance-oriented evaluation Results oriented management Rapid feedback evaluation Performance monitoring Service delivery assessment

Wholey’s Theory of Social Programming Social problems solving through incremental program improvement Culmination of many small improvements eventually results in an improved ability to solve social problems Largely a theory of management rather than social policy and social change

Wholey’s Theory of Knowledge Construction Essentially logical positivism— observations of program occur through operationalizing dependent variables Pragmatism—truth depends on what works in practice (as decided by management) – Knowledge claims are valid if practical consequences satisfy decision makers

Wholey’s Theory of Valuing Valuing is a decision to be made by program management Value claims are a function of performance measurement, comparison of performance to goals, and the use of performance information for policy-making and program management Criteria of merit are defined by management

Wholey’s Theory of Knowledge Use Favors instrumental use Rapid assessment methods provide immediately usable information about programs Instrumental use can occur in the absence of rigorous methodologies

Wholey’s Theory of Evaluation Practice Sequential purchase of information – Evaluability assessment – Rapid feedback evaluation – Performance monitoring – Intensive evaluation Encourages flexibility in applying the sequential purchase of information – In sequence and methodology

“I like the evaluator more in a role of civil servant than of civil philosopher” — Robert E. Stake

Biographical Sketch Ph.D. in Psychometrics from Princeton University M.A. and B.A. in Mathematics from University of Nebraska Emeritus Professor of Education and Director of the Center for Instructional Research and Curriculum Evaluation at the University of Illinois Recipient of numerous awards and honoraria Published numerous influential books, journal articles, and reports

Stake’s View of Evaluation Unlike prior (and later) theorists, developed his theory on the premise of “discovery” – Implied that the only realities (i.e., truths) are those that are constructed through a process of discovery Valuing is socially constructed – The value of a program is best determined by those who experience it – Emphasis was on local stakeholders values Evaluation should be responsive rather than preordinate

Stake’s Influence Developed widely adopted methods for conducting case studies Widely considered a pioneer of qualitative methodologies Responsive evaluation Recognition of and greater concern for localized evaluation with an emphasis on improving local practice

Stake’s Major Contributions Naturalistic generalization Case study methodology Responsive evaluation Naturalistic inquiry Direct influence on constructivist conceptualizations of epistemology and ontology

Stake’s Theory of Social Programming Local heterogeneity is of vital importance as incremental changes should be initiated by local stakeholders to meet their own needs Rejects federal solutions to social problem solving—local problem solving is preferable

Stake’s Theory of Knowledge Construction Responsibility for what constitutes valid knowledge should be given to lay persons (popularization) rather than professionalization (knowledge is the purview of scientists) Denies that evaluators have a privileged position in constructing knowledge claims

Stake’s Theory of Valuing Singular value claims do not exist The value of programs is different for different persons, for different purposes Advocates descriptive valuing by local stakeholders

Stake’s Theory of Knowledge Use Emphasizes enlightenment rather than instrumental or other types of use Use occurs when vicarious experiences that are intuitively understood occur Practical versus scientific use

Stake’s Theory of Evaluation Practice Responsive evaluation and case studies are logically distinct since case studies can be done without doing responsive evaluation – One can do responsive evaluation without doing case studies The choice of responsive evaluation or case studies is defined by purpose

Encyclopedia Entries Cronbach, Lee J. Design, Evaluation External Validity Generalization Logic Model Performance-Based Monitoring Performance Indicator Planning, Evaluation Program Logic Program Theory Qualitative-Quantitative Debate in Evaluation Rossi, Peter H. Theory-Driven Evaluation