EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

Agenda Evaluation theory Standards for program evaluations Background for assessing evaluation approaches Psuedoevaluations Brief activity

Evaluation theory

General features of evaluation theory Unlike social science theories, which are – A set of interrelated constructs, definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting phenomena

Evaluation theories – Describe and prescribe what evaluators do or should do when conducting evaluations – They specify such things as evaluation purposes, users, and uses, who participates in the evaluation process and to what extent, general activities or strategies, method choices, and roles and responsibilities of the evaluator, among others – Largely, such theories are normative in origin and have been derived from practice rather than theories that are put into practice

Theory’s role Has contributed to continuous development of evaluation – Many positive developments – Many negative developments Allows for a large variety of diverse, often conflicting ideologies

Research on evaluation Any purposeful, systematic, empirical inquiry intended to test existing knowledge, contribute to existing knowledge, or generate new knowledge related to some aspect of evaluation processes or products, or evaluation theories, methods, or practices

Shadish, Cook, and Leviton’s criteria for theories of program evaluation Social programming Knowledge construction Valuing Use Practice

Miller’s standards for empirical examinations of evaluation theories Operational specificity Range of application Feasibility in practice Discernable impact Reproducibility

Standards for program evaluations

The need for standards So that members of a profession provide competent, ethical, and safe services To ensure high-quality services and protect the interests of the public So that evaluators deliver sound, useful evaluation services

Existing standards Joint Committee’s The Program Evaluation Standards The American Evaluation (AEA) Association’s Guiding Principles for Evaluators U.S. Government Accountability Office’s (GAO) Government Auditing Standards

Joint Committee standards Utility Feasibility Propriety Accuracy Evaluation accountability

AEA guiding principles Systematic inquiry Competence Integrity/honesty Respect for people Responsibilities for general and public welfare

GAO (general) auditing standards Independence Professional judgment Competence Quality control and assurance

Background for assessing evaluation approaches

Why study evaluation approaches? To help evaluators and clients identify, avoid, or expose misleading or corrupt studies There is no one ‘right way’ of conducting evaluation – To understand the strengths and weaknesses of evaluation approaches and the circumstances under which each is appropriately applied

Pseudoevaluations

Background Pseudoevaluations sometimes falsely characterize constructive efforts—such as providing evaluation training or capacity building—as sound evaluation Some are conducted for corrupt, hidden purposes Others are motivated by political or profit motives In general, these approaches threaten the credibility and integrity of evaluation

Approach 1: Public relations studies Use data to convince constituents that a program is sound and effective Avoid gathering or releasing negative findings Typically use surveys from biased samples Only positive findings are disseminated

Approach 2: Politically controlled studies Conducted with the intent to – Withhold the full set of findings from right-to-know audiences – Violate agreements to fully disclose findings – Selectively release findings Often, decisions are ‘predetermined’ and the evaluation is used as justification

Approach 3: Pandering evaluations Caters to predetermined evaluation conclusions Places evaluator in a favored position to conduct future evaluations Ultimately, both the client and evaluator act together in producing ‘favorable’ findings

Approach 4: Evaluation by pretext Client intentionally deceives evaluator as to true intent of the evaluation – Note: It is the evaluator’s responsibility for investigating the ‘true’ intentions of the client Often emphasizes negative aspects of a program (e.g., so that information could be used to ‘improve’ a program) to support a client’s predetermined decisions

Approach 5: Empowerment under the guise of evaluation Cedes authority of evaluation to groups likely not competent to conduct sound evaluation Gives power and authority to such groups to prepare evaluation reports while claiming that an independent evaluator prepared them Essentially, the approach is directed toward helping groups gain ‘power’

Approach 6: Customer feedback evaluation Supposedly independent ratings and reviews of products and services provided by consumers Whether products and services were actually purchased by those providing ratings and reviews is often unknown Often the ‘vendor’ manufactures ratings and reviews to increase sales

Activity

Discuss the rationale that supports the following statement: “Evaluators should not lend their name and endorsement to evaluations presented by their clients that misrepresent the full set of relevant findings”

Encyclopedia Entries Bias Causation Checklists Chelimsky, Eleanor Conflict of Interest Countenance Model of Evaluation Critical Theory Evaluation Effectiveness Efficiency Empiricism Independence Evaluability Assessment Evaluation Use Fournier, Deborah Positivism Relativism Responsive evaluation Stake, Robert Thick Description Utilization of Evaluation Weiss, Carol Wholey, Joseph