EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

Agenda Introduction to evaluation What are appropriate objects of evaluations and related subdisciplines of evaluation? Are evaluations enough to control quality, guide improvement, and protect consumers? Evaluation as a profession and its relationship to other professions What is evaluation? How good is good enough? How bad is intolerable? How are these questions addressed? What are performance standards? How should they be applied? Why is it appropriate to consider multiple values? Should evaluations be comparative, noncomparative, or both? How should evaluations be used? Why is it important to distinguish between informal evaluation and formal evaluation? How do service organizations meet requirements for public accountability? What are the methods of formal evaluation? What is the evaluation profession, and how strong is it? What are the main historical milestones in the evaluation field’s development?

Today’s learning objectives The distinction between formal and informal evaluation The potential contributions and limitations of formal evaluation Evaluation as a profession and its relationship to other professions Conceptual and operational definitions of evaluation Key criteria for evaluating programs, including merit and worth The roles of values clarification and setting standards in reaching evaluative conclusions Four main uses of evaluation Distinctions between formative evaluation and summative evaluation Distinctions between research and evaluation Historical milestones in the development of professional evaluation

Introduction to evaluation

Evaluation is the single most important and sophisticated cognitive process in human reasoning and logic Permeates all aspects of human activity and has important implications for maintaining and improving services in all areas of society Ultimately, a means for making informed, credible, defensible, non- arbitrary decisions

What are appropriate objects of evaluations and related subdisciplines of evaluation?

Objects of evaluation are referred to as ‘evaluands’ In personnel evaluation the term ‘evaluee’ is used Subfields include, but are not limited to (Scriven refers to these as the seven ‘P’s): – Product evaluation – Personnel evaluation – Policy evaluation – Performance evaluation – Proposal evaluation – Program evaluation – Portfolio evaluation

Program evaluation typically receives the greatest amount of attention Scope of application is nearly infinite: – Educational evaluation – Evaluation of development and aid – Consumer product evaluation – Hospital evaluation – Drug testing – Human and social services evaluation – Science policy evaluation – Etc.

Are evaluations enough to control quality, guide improvement, and protect consumers?

For evaluations to make a positive difference, policymakers, regulatory bodies, service providers, and others must obtain and act responsibly on evaluation findings The production and appropriate use of sound evaluation constitutes one of the most vital contributors to strong services and societal progress

Evaluation as a profession and its relationship to other professions

As a distinct profession, evaluation is supportive of all other professions and in turn is supported by many of them – No profession could exist without evaluation It now also has the ‘trappings’ of a unique, specialized (trans)discipline – Graduate education programs – Scholarly journals – National associations – Professional conferences – Accredited standards – Guiding principles

What is evaluation?

e·val·u·a·tion noun “…the act or process of determining the merit, worth, or significance of something or the product of that process” — Scriven

e·val·u·a·tion noun “…evaluation is the systematic assessment of the worth or merit of an object” — Joint Committee on Standards for Educational Evaluation

Evaluation’s root term, value, denotes that evaluations essentially involve making value judgments Accordingly, evaluations are not ‘value-free’ They should reference pertinent ‘values’ (i.e., qualities or characteristics considered important in a given context)

A note on merit and worth Merit (synonymous with ‘quality) concerns the intrinsic value of an object, independent of context and costs Worth (synonymous with ‘value’) concerns the extrinsic value of an object, considering both context and costs

Needs and needs assessment Needs (something without which satisfactory functioning cannot occur) and needs assessments are one source of ‘values’ used for evaluation and determining the merit and worth of evaluands – Can be categorized as instrumental needs and treatment needs

e·val·u·a·tion noun An extended, ‘values-oriented’ definition… “…the systematic assessment of an object’s merit, worth, probity, feasibility, safety, significance, and/or equity” — Stufflebeam & Coryn

Evaluation should not be confused with… Assessment Review Analysis Applied research Audit Diagnosis And, many other terms that are sometimes used synonymously

How good is good enough? How bad is intolerable? How are these questions addressed?

This process requires clearly defined ‘decision rules’ (e.g., standard setting) and using all relevant ‘criteria’ (i.e., the characteristics defining a good or valuable object of a certain type in a particular context) There are no ‘universal,’ ‘standardized’ methods for constructing evaluative conclusions ‘Beyond a reasonable doubt’ (as used in jurisprudence) has often been used as a ‘standard’ for warrant supporting claims of merit or worth

What are performance standards? How should they be applied?

Evaluation is sometimes characterized as comparing a performance to a standard This process involves setting one or more cut scores against which the performance of something is judged – These methods are typically either ‘norm referenced’ or ‘criterion referenced’

Why is it appropriate to consider multiple values?

Very often different ‘stakeholder groups’ have competing and conflicting values One of the primary tasks of an evaluator is to determine how to adjudicate these conflicts, with professional integrity and careful attention to ethical and moral implications Many, many methods have been put forth for addressing this problem

Should evaluations be comparative, noncomparative, or both

In general, the question of whether an evaluation should be comparative or noncomparative depends on how the evaluation will be used – Comparative evaluations are required when decisions between competing alternatives are required – Noncomparative evaluations are required when a particular course of action has already been determined

How should evaluations be used?

Formative evaluations for improvement Summative evaluations for accountability Evaluations to assist dissemination efforts Evaluations to foster enlightenment Other types of uses include, but are not limited to: – Instrumental use – Process use

Why is it important to distinguish between informal and formal evaluation?

Informal evaluation is what could be considered ‘everyday evaluation’ – Often not systematic and biased in known and unknown ways Formal evaluation should be systematic and meet professional standards – Conducted by those having appropriate competency

How do service organizations meet requirements for public accountability?

High quality efforts to improve services and other types of efforts can be accomplished through: – Accreditation – Internal evaluation – External, independent evaluation

What is the evaluation profession, and how strong is it?

Many technical books, specialized Web sites, discussion groups, blogs, and listserves, master’s and doctoral programs, institutes and workshops on specialized evaluation topics (for example, the Evaluators’ Institute, client organizations that fund a wide range of evaluations; evaluation companies, guiding principles for evaluators, and standards for program, personnel, and student evaluations) These are substantial gains considering the field’s status in 1964, when it had none of these elements The evaluation field is still immature, however, when compared with established professions, such as medicine

What are the main historical milestones in the evaluation field’s development?

The pre-Tylerian period: Developments before 1930 – Initial use of standardized tests as measures of educational quality The Tylerian period: – Development of objectives-based methods for evaluating educational programs The age of innocence: – Very little evaluation requirements; most were internal, self-evaluations The age of realism: – Elementary and Secondary Education Act (ESEA) issued, requiring evaluation of all federally-funded programs and policies, Title I, etc. The age of professionalism: – ‘Identity crisis’ among evaluators and there roles The age of global and multidisciplinary expansion: 2005-present – Professionalization of the field

Encyclopedia Entries Bias Causation Checklists Conceptual Use Consumer Effectiveness Efficiency Epistemology Evaluation Use Experimental Design Experimental Society Impartiality Independence Instrumental Use Intended Uses Judgment Merit Modus Operandi Ontology Outcomes Paradigm Positivism Postpositivism Process use Quantitative Weight and Sum Recommendations Synthesis Transdiscipline Treatments Worth