EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2011

Agenda Course overview Introductions Activity 1 Brief lecture introducing evaluation theory – Covered in greater detail in coming weeks Questions and discussion

Course Overview This course is designed to provide an overview of the practice and discipline of evaluation – Comparative study of theory, research, and practice perspectives – Analysis of core concepts and definitions, rationale and uses, the field’s history and standards, alternative models and approaches, and emerging and enduring issues

Course Overview The website for this course is located at foundations-of-evaluation/ From this site you can access – The course syllabus – Supplementary readings – Weekly lecture notes – Other materials related to the course

Course Overview Required textbooks – Alkin, M. C. (Ed.). (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage. – Mathison, S. (2005). Encyclopedia of evaluation. Thousand Oaks, CA: Sage. – Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Thousand Oaks, CA: Sage.

Course Overview Primary learning objectives – A deep understanding of a wide array of evaluation theory and practice perspectives – An in-depth understanding of the origins and history of evaluation as well as its evolution toward an independent discipline – A clear understanding of key evaluation concepts/vocabulary/terminology – A clear understanding of the nature and purpose of evaluation, and the distinctions between evaluation, basic and applied research, and related terms such as assessment and diagnosis

Course Overview Primary learning objectives – An ability to describe, distinguish among, and critically evaluate the usefulness and validity of selected models and approaches to evaluation, and to identify the conditions under which each should be used – A firm grasp of the fundamental logic and methodology of evaluation – A basic understanding of how to integrate traditional methodologies with evaluation- specific methodologies – An evaluative and critical thinking mindset, in general

Course Overview Secondary objectives – Conveying constructive criticism in a professional, balanced, and tactful manner – Facilitating discussion to engage others in dialogue about evaluation theory, method, and practice – Writing clearly and concisely for both academic and non-academic audiences – Giving high quality, professional oral presentations for both academic and non- academic audiences

Course Overview Course components – Attendance & class participation 10% – Critical readings papers 30% – Application paper 20% – Thought paper & presentation 20% – Final examination 20%

Course Overview Schedule of topics – Introduction to evaluation theory – Foundational evaluation concepts – Stage one theories & theorists – Stage two theories & theorists – Stage three theories & theorists – Methods-oriented theories & theorists – Valuing-oriented theories & theorists – Use-oriented theories & theorists

Course Overview Weekly meeting structure – If necessary, a question-and-answer session for prior week’s material (30 minutes) – Lecture (60-90 minutes) At the end of each lecture is a list of entries from the Encyclopedia of Evaluation that you are expected to study (many will appear in the final examination) – Discussion (30 minutes) – Activity (30-45 minutes)

Introductions Who are you? – Why are you here? – What do you expect to learn? – What prior experiences do you have with evaluation? Research? Design? Measurement? Statistics (or analysis in general)? Who are we?

Activity 1 Draw the first image that comes to mind when you hear the word evaluation (15 minutes) In small groups share your images, identify common themes, and write your themes on the flipchart (20 minutes) Share your group’s themes (15 minutes)

Meta-Theory and Theory A meta-theory is a theory whose subject matter is some other theory – In other words, it is a theory about a theory A theory is a set of interrelated constructs, definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting phenomena

Evaluation Theory Evaluation theories describe and prescribe what evaluators do or should do when conducting evaluations and are mostly normative in origin They specify such things as evaluation purposes, users and uses, who participates in the evaluation process and to what extent, general activities or strategies, methods choices, and roles and responsibilities of the evaluator, among others

Classification of Theories Shadish, Cook, and Levition’s five principles that undergird evaluation Alkin and Christie’s evaluation theory tree that classifies and describes major theorists’ orientation Stufflebeam and Coryn’s five categories Scriven and Fournier’s more general ‘logic of evaluation’

Program Evaluation Theory Shadish, Cook, and Levition’s five principles that undergird evaluation – Theory of practice – Theory of knowledge – Theory of valuing – Theory of use – Theory of social programming

Evaluation Theory Tree Alkin and Christie’s theory tree

Broad Category Classification Stufflebeam and Coryn’s classification system for evaluation models and approaches – Pseudoevaluations – Questions- and methods-oriented – Improvement- and accountability- oriented – Social agenda/advocacy – Eclectic

Logic of Evaluation Scriven’s logic of evaluation is the closest to a meta-theory of evaluation 1.Establishing criteria On what dimensions must the evaluand do well? 2.Constructing standards How well should the evaluand perform? 3.Measuring performance and comparing with standards How well did the evaluand perform? 4.Synthesizing and integrating information/data into a judgment of merit or worth What is the merit or worth of the evaluand?

General Premises 1.Factual premises – The nature, performance, or impact of an evaluand or evaluee – Roughly equivalent to description (“what’s so?”) 2.Value premises – The properties or characteristics (i.e., criteria and standards) which typify a good, valuable, or important evaluand or evaluee of a particular class or type in a particular context

Value Premises 1.General values – The merit-defining criteria by which an evaluand or evaluee is evaluated; the properties or characteristics which define a ‘good’ or ‘valuable’ evaluand or evaluee 2.Specific values – The standards (i.e., levels of performance; usually an ordered set of categories) which are applied and by which performance is upheld, in order to determine if that performance is or is not meritous, valuable, or significant

General Logic

Working Logic

Encyclopedia Entries Assessment Accountability Auditing Campbell, Donald T. Cook, Thomas D. Criteria Evaluand Evaluation Evaluation Theory External Evaluation Formative Evaluation History of Evaluation Independence Logic of Evaluation Objectivity Scriven, Michael Shadish, William R. Standards Summative Evaluation Value-free Inquiry Value Judgment Values