1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

2. Agenda Evaluation theory Standards for program evaluations Background for assessing evaluation approaches Psuedoevaluations Brief activity

3. Evaluation theory

4. General features of evaluation theory Unlike social science theories, which are – A set of interrelated constructs, definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting phenomena

5. Evaluation theories – Describe and prescribe what evaluators do or should do when conducting evaluations – They specify such things as evaluation purposes, users, and uses, who participates in the evaluation process and to what extent, general activities or strategies, method choices, and roles and responsibilities of the evaluator, among others – Largely, such theories are normative in origin and have been derived from practice rather than theories that are put into practice

6. Theory’s role Has contributed to continuous development of evaluation – Many positive developments – Many negative developments Allows for a large variety of diverse, often conflicting ideologies

7. Research on evaluation Any purposeful, systematic, empirical inquiry intended to test existing knowledge, contribute to existing knowledge, or generate new knowledge related to some aspect of evaluation processes or products, or evaluation theories, methods, or practices

8. Shadish, Cook, and Leviton’s criteria for theories of program evaluation Social programming Knowledge construction Valuing Use Practice

9. Miller’s standards for empirical examinations of evaluation theories Operational specificity Range of application Feasibility in practice Discernable impact Reproducibility

10. Standards for program evaluations

11. The need for standards So that members of a profession provide competent, ethical, and safe services To ensure high-quality services and protect the interests of the public So that evaluators deliver sound, useful evaluation services

12. Existing standards Joint Committee’s The Program Evaluation Standards The American Evaluation (AEA) Association’s Guiding Principles for Evaluators U.S. Government Accountability Office’s (GAO) Government Auditing Standards

13. Joint Committee standards Utility Feasibility Propriety Accuracy Evaluation accountability

14. AEA guiding principles Systematic inquiry Competence Integrity/honesty Respect for people Responsibilities for general and public welfare

15. GAO (general) auditing standards Independence Professional judgment Competence Quality control and assurance

16. Background for assessing evaluation approaches

17. Why study evaluation approaches? To help evaluators and clients identify, avoid, or expose misleading or corrupt studies There is no one ‘right way’ of conducting evaluation – To understand the strengths and weaknesses of evaluation approaches and the circumstances under which each is appropriately applied

18. Pseudoevaluations

19. Background Pseudoevaluations sometimes falsely characterize constructive efforts—such as providing evaluation training or capacity building—as sound evaluation Some are conducted for corrupt, hidden purposes Others are motivated by political or profit motives In general, these approaches threaten the credibility and integrity of evaluation

20. Approach 1: Public relations studies Use data to convince constituents that a program is sound and effective Avoid gathering or releasing negative findings Typically use surveys from biased samples Only positive findings are disseminated

21. Approach 2: Politically controlled studies Conducted with the intent to – Withhold the full set of findings from right-to-know audiences – Violate agreements to fully disclose findings – Selectively release findings Often, decisions are ‘predetermined’ and the evaluation is used as justification

22. Approach 3: Pandering evaluations Caters to predetermined evaluation conclusions Places evaluator in a favored position to conduct future evaluations Ultimately, both the client and evaluator act together in producing ‘favorable’ findings

23. Approach 4: Evaluation by pretext Client intentionally deceives evaluator as to true intent of the evaluation – Note: It is the evaluator’s responsibility for investigating the ‘true’ intentions of the client Often emphasizes negative aspects of a program (e.g., so that information could be used to ‘improve’ a program) to support a client’s predetermined decisions

24. Approach 5: Empowerment under the guise of evaluation Cedes authority of evaluation to groups likely not competent to conduct sound evaluation Gives power and authority to such groups to prepare evaluation reports while claiming that an independent evaluator prepared them Essentially, the approach is directed toward helping groups gain ‘power’

25. Approach 6: Customer feedback evaluation Supposedly independent ratings and reviews of products and services provided by consumers Whether products and services were actually purchased by those providing ratings and reviews is often unknown Often the ‘vendor’ manufactures ratings and reviews to increase sales

26. Activity

27. Discuss the rationale that supports the following statement: “Evaluators should not lend their name and endorsement to evaluations presented by their clients that misrepresent the full set of relevant findings”

28. Encyclopedia Entries Bias Causation Checklists Chelimsky, Eleanor Conflict of Interest Countenance Model of Evaluation Critical Theory Evaluation Effectiveness Efficiency Empiricism Independence Evaluability Assessment Evaluation Use Fournier, Deborah Positivism Relativism Responsive evaluation Stake, Robert Thick Description Utilization of Evaluation Weiss, Carol Wholey, Joseph